Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS

Hey, welcome. Welcome to Beyond Done. Uh, we're gonna get started here with this workshop. We got a lot to pack in, and, uh, we wanna make sure that we have time for questions and, um, uh, getting, getting, like everyone situated. So we're gonna go ahead and get started. So what we mean by Beyond Done, just to like jump straight into it, um, you probably saw a tweet about this incident that happened a while back, uh, where an agent just started deleting production

databases, and in nine seconds everything was gone. Um, that is what we want to help you prevent here today, uh, while still trusting your agents and not having to look at every line, even look at every PR. Like, this is where we think things are going and how do we get there? How do we make this a reality where instead of having that broken piece, we can have it calm, have, uh, everything work and let you, to some degree, trust your agents, but build that trust over time.

I'll t- say also that it's not just about, uh, preventing errors, right? But this is also what we're gonna look at today, foundational to achieving true scale with, with agents as well.

So I'm Nick Nisi. Uh, I'm an engineering manager at WorkOS. Uh, I live in Omaha, Nebraska, and, uh, I've been there for about two years, and I do a lot with agents. Uh, I haven't written code since July 30th, 2025. I know 'cause that's the day I got back from Laracon, and I saw PHP and I was like, "Never again." And, uh- ... here I am. Uh, and with me is my best friend, Zack.

Hey, I'm Zack Proser. I'm, uh, an engineer on the Applied AI team at WorkOS. Super excited to be here with you all today in New York.

Now, uh, we had this up before. This repo is your workbench. Um, all of the slides are in this repo. You can get it from here. You can also view it at, um, workos.cloud/beyond-done. Uh, I know that's a mouthful, but the next PR, or the next QR code I believe will take you there. Um, sorry, talks.workos.cloud/beyonddone. Uh, and I just wanted to call it out because on that, I'll switch to it real quick, um, all of the slides are out there.

There's also a page for the workshops, like the, the exercises that we're going to go through. And so for each piece of it, you can go through it, you can go through it on your own time, you can go through it later. This is a, a step-by-step guide to go through that. There's also a glossary, uh, covering all of the terms, big and small, uh, some that you're familiar with, uh, hopefully most that you're familiar with. But if there's any that you're not, uh, you can talk in here. Uh, or, sorry, you can look at the, the glossary for those terms. There's also an ask anything button,

and you can talk to a RAG that's trained on this, uh, slide deck and, uh, hopefully get your answers there. But you can also just talk to a human, us.

Indeed. Yeah. We'll leave, leave this up as well. We find that a lot of folks want to ask questions and for whatever reason can't, so be sure to check that out and use that.

Yep. Cool. Give it just another second. I see some cameras pointed at QR codes. Security would be so mad at us.

All right. Um, so when we're thinking about, uh, being done with work, uh, agents being done, uh, there's lots of things that go into us trusting that. And easily the first one is just, is the code right? Is the code what I would have written? Did it generate a good PR? And I go through that like a normal, normal code reviewer. Um, but that takes a long time, and these agents can write code a lot faster than us, so we don't want to

rely on that too much because it's, uh, it just... There, there's a, a signal-to-noise ratio that starts coming into play, and we just have too much code to review. We're not writing any code anymore, and, uh, we're less familiar with that code, so we become further abstracted away from it. So we need to find other ways to do that. And there's plenty of ways that you can do that as well. There's, um, you know, using tools like, uh, browser tools like Playwright, uh, for example, to say like, "Hey, I have this website, and I submitted a form. The submit worked,

I checked that, uh, and it showed me a success message." That's a great test to write. Not even really anything agentic about it, but it is something that agents can use to verify their work. Um, and they can self-report that, and you can trust that, uh, oftentimes, but not all the time. And so that's really what we wanna drive is like, how do we make it so that we can actually trust these reports?

Absolutely. So, uh, everything that we've got here for you today in this workshop, as Nick mentioned, is backed by a repo, and it is interactive. So be... You know, feel free to take an opportunity to actually run this, to experiment with it, uh, to take something back that you might, you know, uh, use at work or at least, uh, experience in a different way.

Yep. Um, in this, so if, like, if you just go into the repo like I'm doing, and you do pnpm snack up. Oop, if I can type. Snack up. Uh, ugh, this is why I have shortcuts for everything. P is my pnpm. Um, it will pop up the browser, and we can look at it, and we can see this Midnight Snack Club I- app that we made. Um, and you can play with it right now, and I can test this myself. Let me just put in my email address.

I'll put in, um, [REDACTED].

Thanks, buddy.

Yep. Uh, and look at that. Success. It saved the email. The email is here. I can see it. If I had a test that was looking to confirm that the email went on the page, uh, it worked great. So that's what these examples look like. You can kind of see it, um, as you're going. Uh, but if you reload that page just like this, nothing's there. So even though... Oops, what did I do?

I'll switch back. Uh, if you reload that, uh, it shows that nobody's here. I can't drive Chrome apparently. Um, and that's because it did update, but it didn't persist that anywhere. So we didn't have any kind of verification. We were just checking UI, seeing, "Oh, did that actually do anything?" And yes, it did do something to the UI, but we didn't check the other half, and so it was never persisted, and we lost that email. So sorry, Zack, you're not signed up.

That's okay with me. Um, yeah, I think this is a really critical point. So, uh, just show of hands, who in this room is, is building, uh, with... A ton with agents, just to start with as a first question. Okay, awesome, like, more than half of the room. Uh, how many folks are running multiple terminal tabs or multiple chat sessions or multiple agents, like everybody that put their hands up the first time, right? And so this is what we're all pr- uh, experiencing, at least what I'm experiencing, what Nick's experiencing. If this is the 18th, you know, lane of work that you're driving today, and you have, like, d- you know, diffracted attention

to actually spend on it, these are the subtle bugs that will creep in because the agent's like, "Yeah, I did what you said. The code looks good. It builds, it runs, the tests pass," and it's still wrong. And so what we're gonna ask and explore in this workshop together is, how do we actually fix this problem so that the agents that we're now all managing as, like, herds of insane token machines, uh, can actually figure out what they need to do themselves and when they've made a mistake?

Yep. And you'll notice that we ar- aren't really looking at the code for this. We're not really gonna look at the code too much for this because, uh, like I said, I saw PHP and I'm scared. So we don't look at code anymore. Um, but all of this builds up to this great quote by, uh, a famous potato on, uh, X call, uh... Her n- her name's Lauren Tan. Uh, and she has this quote, "If any agent..." I- it- sorry, "If an agent can verify its own work, nothing..." Can't verify. "If an agent can't verify its own work, nothing else matters."

Yeah. Another way to say it is, is that, um, and I, I think this is true in, in my experience and others as well, is that we're kind of converging on the point that verification now, especially as model, uh, capabilities advance rapidly, is, is one of the key foundational things to focus on, right? It's like reliability, verification, and evals, especially as we're starting to see more and more harnesses come online that have the concept of RSI or re- you know, recursive self-improvement baked into them. It feels like we're shifting to a world where we may all be maintaining multiple self-evolving

code bases eventually, right? And in that world, how do we set the eval foundation so that the agent actually knows what good looks like and can do those hill climbs on its own? And how are we going to give the agent the feedback it needs to understand that it's failed something?

Yep. Until we can solve that, we're not gonna trust these agents to really accelerate our work to its fullest extent. And, uh, so we need to solve that, uh, in various degrees. And it's not just going from zero to one, but it's a gradient. And getting us closer to one, uh, to where we can fully trust these agents is, uh, is the goal.

Uh, and I'll also just share that, you know, again, anecdotally, this is what we're all experiencing at, at WorkOS. Uh, we've got an applied AI team that's building a ton of different stuff, uh, for other teams and for ourselves, and it is just a night and day experience when you are iterating on a code base that has, you know, the foundational eval in place, a, a baseline that can be run and trusted. Uh, you know, c- we'll get into some of the tips and tricks, but the ability to actually see that it did or did not do something correctly in the same way that we would as human developers, right? So we're, you know, a, a, we've...

The, the metaphor of us all becoming agent harness managers is, is, you know, well-trod. Um, this is how we ensure that each one of them is actually doing correct work and we're not wa- you know, ba- burning money and time essentially.

Yeah. This is the equivalent of saying like, "Make no mistakes," right?

Right.

I- if you wanna have some verification in there, you can't just like guess at what it needs to be. You need to give it clear, defined goals to get to and give it a way to observe and check those goals. If not, uh, the a- agent's gonna do something, and it's gonna spend some tokens, and it might get to the end that you expect it to, uh, but there's no guarantee of that. Every, for every detail that you don't fill in i- in terms of an outcome and what you actually want to get to, it's just going to infer that or make something up, and

that assumption that it makes may not be the assumptions that you make, uh, or that you were expecting because you're a rational, logical human. Uh, and, uh, you, you were just not aligned on that yet. And so, um, you, you need to have these systems in place to allow for it to observe, understand what you want, observe that, repair that, and then recheck that.

Yeah. And I think just, again, anecdotally, like we're all finding that, um, you know, the tired way to do it is to have a giant massive PR and have one agent session that's running for three or four days and say, "Fix this, fix this, fix this. Make it better." And what is working, you know, essentially perfectly for folks is decomposing tasks really cleanly, giving the each task and agent the, in the prompt exactly what it needs to do to know that it is actually done-

Mm-hmm

... and to verify the state of done. And those will just rip. I mean, we have people on our team that are, are 900 PRs a month, um, and, and no instance in like, uh, improving KPIs for the software. So, uh, just anecdotally, it is, um, something that we all kind of have to s- take a step back and think about, "I'm not approaching this, you know, code base anymore in a traditional way. I'm almost gardening multiple code bases and letting them self-evolve."

Yeah.

And this is something that we build and we hold our own tools to as well. This is how Zack and I stay productive at WorkOS and how we get our own work done. Uh, and we've each built kind of our own examples of this and our own ways of, uh, approaching this, and we really think that that's like a good thing right now. It's too early to really calcify on anything, so experimentation is key, finding what sticks, and then running with that. And, uh, one thing that we're both running with is the verification piece or like deriving everything from that verification loop, uh, because, uh, as, as it

says in this quote here for, for mine, uh, "Status comes from records, not model prose." Um, so it's, it's, you know, show me the receipts. Show me the actual tangible proof that it actually did it, uh, what I expect. Don't just tell me, "Yeah, it passes. It looks good." Um, because there's nothing to back that up. I need actual screenshots, videos, test results, smoke tests, um, any, all of the above to prove to me, without me having to look at the code, that it's actually doing what I want. Because at the end of the day, I don't care about the code as much, I care about the result, and

what it's doing, and what value it's delivering. And it can have bad code, honestly, and deliver that result pretty well, uh, or it could have really good code that doesn't r- uh, give me that result at all, and I need to find the right balance in there, which is giving me the result that I want , not the code.

Yep, indeed. Uh, so I've been doing this as well. I, I built a tool that I call Context, or CTX. It started out as a cross agent harness memory layer, 'cause I'm rapidly experimenting, as all of us are, with like ChatGPT, Claude Code, uh, Pi, et cetera, and I wanted them to all be able to write and read from memories, and that started working. And then the more that I was running mul- more, multiple agents and, you know, average day is like 18 or 30 terminal tabs, the more that I realized, uh, it might be an interesting experiment to try to centralize authority for verification in a web service. So the CTX is simple web app, it runs on Cloudflare.

It, it was vibe coded initially. It works. Uh, it started as memory. It was able to hand out secrets to agents without them ever seeing it or getting the credential, and that worked too. And then now as an additional experiment, I'm creating verifiers. And the way that that works is the agents are assigned work, as we're all doing with prompts. They go off and do that work. They come back through the MCP connection to CTX and say, "This is what I've completed. Uh, am I done? Please verify it." So let's say in the example of, um, uh, one of the workflows needs to verify that a PR was

created correctly. And so that is going to run an evaluator on a known verifier. That could just be a simple CLI to start with, as it is in my repo, and then that becomes a hardened piece that you can reuse again and again. It's going to the GitHub API and verifying that that PR truly does exist, and is healthy, and that all the merge checks pass. And th- only then will it mark that node satisfied in the workflow. Otherwise, it will redirect to... It will redirect the agent and say, "These are the things that are still failing." Right? And so again, an experiment, but this is kind of how we've been

trying to play around with this and see which patterns are, are gonna stick and, and what works and whatnot. Um, but it's also just kind of a fun time to do that vibe coding because, uh, I require, uh, weird pixel art across all of my apps. So, uh, you can see in this case there was a simple workflow, and one of the verification gate is that requires that all pull requests were merged. And so that work is done by the centralized authority. It goes out and uses its verifier, checks the API or whatever the artifact is, and then finally delivers the status over here on the r- right sidebar, if you go even down, and

references the actual repository it checked and the PR, right? So the simple example, but you could imagine having standardized verifiers in a library, or in skills, or in tooling that you share amongst your, your org, and this could be one of the ways that you start to build toward accelerating, you know, PRs and, and less and less human oversight.

Mm-hmm.

Uh, and then a similar tool, uh, that I built is called Riker. This is more like a engineering-focused Grok bot or Muse, uh, that's trying to be extremely flexible in how you hand work to it. But then once you hand work to it, its job is to babysit it. And so I don't wanna have to babysit all of the PRs that I'm, I'm bringing to fruition. At some point, there's a, like this level of diminishing returns for me and my focus on, on the work that I'm doing because I end up just like babysitting the PR, you know, fixing the tests, fixing the, the reviews, uh, that are

coming in, the, the review comments and things like that. So I wanna be able to get the idea done of like what we should be doing and how we can verify that that work is done. Uh, and then I can hand it off to Riker. It can go do the work of, you know, making sure that the, the reviews are done, uh, that when it fixes code, uh, from PR reviews, that it's still matching our overall goal of working software, uh, in some way. And then, uh, can just verify that and, and alert me back when it needs me

instead of me having to babysit it. Um, and so with that it's just, um, you know, it builds a check, it figures out what tools it needs to run. It will run them, uh, verify that they all pass. It'll keep that evidence and show it to me in different ways, uh, and then decide when to stop. So if it can verify, yes, it was working, we've now fixed this bug, for example, um, we put them together and it works. Uh, so it's really just keeping me out of having to be in the minutia of

shipping the, the, like final 10%, uh, and just focusing on where I'm valuable as a human, which is why we're doing the work and how we're... and, uh, to an extent, how we're doing it.

Okay. So j- maybe just as a mental experiment or, um, a thought experiment, if you will, let's just kind of a show of hands like, or sh- feel free to shout out, like how would we start to think as a human developer of verifying a web app, like maybe in the before times, before an LLM?

Yeah. What are things that you would do to verify?

And we'll, we'll throw some easy ones up here to get it started. So like refresh the page. Is the application still visible, right?

What's that?

API check.

Absolutely.

Yeah.

Yep.

Yeah, network.

Check the network tab.

Yep.

Uh, see if it's healthy, right? What are the response codes I'm getting back from the API that I expected based on what I sent it?

Yep.

Absolutely.

Local host, uh, or local storage, checking persistence in some way, console logs, um, things like that. Network, uh, like you said, and, um, like reading what's actually on the page-

Sure

... the contents of the page actually-

Interacting with it, right?

Yeah.

And the reason we're, we're saying this is like this is really to set up the, uh, the transition from we were the human developers doing this all the time. This was part of our core responsibility when we're hand-chiseling code, if you will. And then what is the delta between doing this and then in the, the early stages of LLM time saying, "Here's three paragraphs of text. Go execute these instructions." The delta between those two things is the agent's ability to verify that it actually did what you asked it to do in a real

application, and that its changes did not break anything, right? Otherwise, you're just getting the confabulations that LLMs are designed to give you-

Mm-hmm

... 'cause they're, they're trying to process the text, they're trying to make you happy.

Uh, your agent can't do any of this just in a terminal alone. Uh, it needs tools to do that. Uh, fortunately, tools are very easy to add to agents now. Um, and we just have to tell them how to use it, or tell them specifically what we're trying to, to prove. Um, when you do that, you don't want it to be just a paragraph of like, "Hey, do this ver- always verify the change in a browser," whatever that means. That's pretty vague. Uh, "Remember to reload and check," that's a little bit more specific. Uh, check that the data persists. Look at the console for

errors. Uh, getting a little more specific, but, like, that's a lot, um, to put in there, and you can certainly do that. That's what a lot of skills do. That's a lot... what a lot of just, like, saved prompts do. Um, but if you forget anything or things change, that can leave it up to a lot of interpretation. Um, one thing that might be better is to have a script that can prove it out in different ways, um, so that you can just run that over and over, and it's doing more of a deterministic check across a number of things, and then it can feed that output back into the

agent, and it knows exactly what was wrong, uh, and how to fix it quickly, and it just closes that loop so that it's a much faster, uh... and the agent can just spin that, run it again, check, verify, run again, and just do that over and over until there's no more errors.

It's also uniform if you think about it, right?

Mm-hmm.

Um, we know that LLMs are non-deterministic, so even giving 30 agents the exact same prompt, y- you have a potential for variance in your outcomes. And pinning it down to a single command that is backed by actual bash commands and bash calls, right? Like, that's, that's significantly more deterministic. And anecdotally, the folks, uh, like Lauren Tan and others that are achieving thousands of PRs ostensibly a month and, uh, building ambitious products that were not possible to build in that time before, have mentioned that this is the first thing that they c- they concentrated on that actually unlocked the velocity,

right? So if your specific web app, for example, does not already have a bespoke CLI that allows any agent to fully flex the functionality that you care about verifying, then you're going to have a much rougher time and an unpaved path if you're using prompts, you know, alone, uh, compared to actually, again, giving each agent the ability to say, "Yes, I actually built that feature. It is actually performant. It didn't cause any errors, and the application looks correct."

So for this workshop, uh, we made a little tool. Uh, we've been running it a couple of times, but it's called Snack. Uh, and you can just run it with pnpm snack in that repo. Uh, it's not meant to be, like, a reusable tool, but just kind of like a template for how you could set up this, uh, specific for the applications that you're working on, and then use that as a way to tie into the LLMs that are actually working on that application with you. And so Snack can do a couple of things. It can, uh, manage the health of the application. It can bring it up, it can take it down. It can run a doctor command to see is it actually... you know, is the database

running? Does it have network connection? Et cetera. Um, that quick dry or that quick check that it can do. Uh, then it has, uh, drives, so you can tell it to join. Remember, this is our, our, uh, Midnight Snack wait list. Uh, so it can join the wait list, it can reload the page, it can reset all of the data so it can't find it. You can have it look up things, so take a snapshot, uh, look at the network storage, look at console logs, um, look at local storage, uh, screenshot it, run evals, all, all of

the above. Uh, and then finally, it can prove it in two ways. It can actually run a prove command where it's checking for deterministic output of that, and a judge, which is using a System 1 model to, uh, in... like, get an idea of whether we're going in the right direction without it necessarily matching one to one.

Just to, like, fully develop this idea, when you do something like this in a repo that you haven't already had something like this before, the velocity gains that you unlock are not solely for you, right? It's for everyone else on your team that then comes along and is also able to issue significantly more PRs significantly faster and significantly fa- uh, safer, right? More safely, so that, um, you know, everybody on the team actually benefits. And a- again, that's like why w- we keep harping on this being like a foundational concept now.

Yeah. Uh, and if you think about it, like, that's a really big piece of... I- it might even be more important than your software, uh, than the actual application, is the way that you verify it, the way that you test it, uh, the way that you prove that it works because that is what you could use potentially, like, uh, a lot of people are doing in converting to Rust. Um, you could take your app, uh, like the Bun team did, convert it to Rust, uh, Go... TypeScript, convert it to Go, uh, not necessarily with the same level of LLM help, but,

um, they did use LLMs to help with that. Um, and then I think Theo just last week converted it to Rust again.

Mm-hmm. Yep.

So uh, when you want that, if you have all of those verifiers in place, then your source code doesn't matter as much. You can write it again and prove that it works. Keep going.

Absolutely.

And so, um, these tools give your agents their eyes, or y- if you think about it, like, the tools that your agent use, that gives it the body. The LLM is the brain, and then the way that it reaches out and interacts physically, uh, virtually on your computer is through these tools. And so things like, uh, join is, like, actually having it run and do things, and then it can check and see, you know, did I... did that, that actually succeed in some way? Uh, check storage, run l- reload, uh, snapshot, et cetera.

And so let's just run a couple of these, uh, and see. p snack up. That brought up our, um, browser, and then I'll open up another browser or another terminal. Too many terminals. And run, um, p snack snapshot.

And that gave us a snapshot of the page in, as JSON that we can see. If I put, um, Zack @zack.zack, uh, and add that to the page, hopefully we'll see some Zacks in there. There we go. So that's a way that we can verify. Uh, we can also do screenshot to make it a little easier. That should pop up a screenshot.

And yeah. So that gave us the ability to, for example, not just like pass this to the LLM and have it do interpretation on it, but like proving back to us, that's the end goal, right, is proving back to the human. Being able to take screenshots, record videos, put those on the pull request. You can do that with the, the GHCLI now, um-

Chrome debug- debugging tools.

Chrome debugging tools, yeah

Like flame, flame graphs. Where is the actual bottleneck? Don't guess and, you know, spin your wheels for three hours. Like, actually figure out which part was causing the render to be slow.

Yeah. So the real key here is that there's no, um... It's, it's not just code that you can look at to interpret whether this, this application is running correctly, but you can build tools that will help y- prove to you that it's working correctly without you having to read the code at all. Uh, because this is all we care about. Is this what a user sees? Is it correctly rendering in this browser at this width with dark mode, with light mode, et cetera? And you can run infinite tests on that and run them over and over and over in a loop, uh,

which as Zack said, unlocks your whole team to move faster because it... you've given the agent the ability to run those checks itself as well.

So many questions.

So if we run this, uh, if we ran it just like this where we ran the doctor, uh, and then we add, uh, Ada @example.test, and then we check the storage, it's there. We run... Uh, if we check the storage, it's not there, um, we would see local storage is empty, and if we re- run reload and then snapshot, we would verify that it is in fact empty. So this is a way that the tool itself could run in a loop, could check all of these things, and yes, confirm with... before it even gets back to us, "Oh, I see that we're not actually persisting this.

I forgot to do that part," and the agent can just move on from there. Um, real quick, w- we wanted to stop and take a, a few questions, if there are any. Um-

Yes, sir.

Yeah. Uh, so the question is, uh, are we, uh, using models to write JSON, uh, to, to then like help us- To make our code deterministic. To make our code deterministic. Yeah. To bring that back. Um, the answer for me is yes. Uh, I'm, I'm using that, I'm running it, and then I use, uh, like I'll, I'll use unit tests around that to verify in different ways, catch edge cases, uh, and then verify that the code... That, that, that is like a real

interpretation or real representation of what I'm expecting to get back from like a System 1 model, uh, and use that to prove that, that I'm on the right track. But then have... keep those as well to kind of like have a, another check, uh, to make sure that it doesn't drift over time or things change.

The, the, the general process that we're talking about? Yeah. So when... what I'm talking about, uh, at least on our team, we, we build a ton of applications that are f- internal tooling. We have some customer facing applications as well. They tend to be, uh, TypeScript, and they tend to be deployed to Cloudflare. So they're, you know, they're full stack applications. It works perfectly. You could even have evals that are focused on the back end or database operations, right? You could get as granular as you want with it. Um, but it's absolutely, yeah, works full stack.

Yeah.

And, and I expect that the way that we're all gonna continue interfacing with software in the future is just gonna be kind of like the gardeners making sure that they're evolving correctly-

Yeah

... on their own. Yeah.

That's why w- uh, we're not like giving out a tool because we couldn't give you a, a tool at this time that works generically for every single application that everyone in this room is working on. You kind of have to build up the tools that fit the app that you're working on, and then give that to the models, and just make sure that they know how to run them well. Um, as a, a-

Oh, s- sorry. I'll, I'll also say like your evals, what you actually-

Yeah

... need or care about as a team or a company or business are gonna wildly differ from, you know, if you're in like the interior design space versus some other place where like speed is the only thing that matters. Um, I, I think you're gonna have like wildly different evals, and there's a value to building those up with your team.

Yeah.

Yeah.

And there's, there's lots of different ways to get the data to the, like to the model in different ways for... so that it can interpret it and it can keep up, and that's like a, a huge piece of like keeping the model in the loop a- as to what's going on. Uh, the Amp team has a really great post on it. I think it's called... I can't remember the, the post, but I'll get it out there. I think it's called Loop Driven Develop... Uh, no. I can't remember. But, uh, what they... In, in the example, they show like a canvas with a circle and a ball bouncing inside of this circle, uh,

and then like there's a little slit where the ball, if it hits just right, it can fall out of the circle, and then the game, the game is over. And, uh, the whole idea of that is making things feedback loopable. That's what it is. Um, where, you know, the model can't actually see the running canvas and where things are going, but if you, as a very contrived example, if you just print out the coordinates of that ball at every point, and you see a bug where, you know, once it gets to this specific point, it jumps to the other side of the screen. Well, if it's moving more than one pixel at a time, like per loop,

something is wrong potentially, and so there's a bug in there. If you can surface that to the model, then it can help you debug it, where it doesn't actually have eyes to see the canvas and see the ball not behaving well, like with physics. So-

And a- also, once you do that, you can... you're also then free to spin up 30 different variants of software that are, you know, speaking to that eval, and then pick the one that you want.

Yeah.

Um, so that's the other thing that un- unlocks, right? It's really like foundational for just going... And especially if we expect agents to impr- or models to improve, you know, tokens to get cheaper potentially, um, then, you know, you could imagine that we're all going to be like coming in on Monday and saying like, "I would like to see 35 versions of this new feature or application here-

Yep

... and pick one."

And so how do we actually run or create that? Well, in this case, like we said, we can't really prove it, uh, for everything, like universally. It really has to be something that's unique to your application. And so for our Midnight Snack page, this is what the proof, uh, the prove command might look like, where it's basically using, uh, a tool like Playwright, um, to check that a, uh, an email was actually saved when it gets on the wait list. So, uh, we can check that it doesn't have any on the wait list,

any emails, then we can fill in an email on the next line, and then we can get the button and click it. ... uh, the join wait list button, and then we can expect that the status, uh, element on the page has a waitlisted in there. Uh, and then check the inner text of that to see if our email address is in there. Um, and so that's one simple way to do it. Very easy if you've ever written, uh, an end-to-end test with like Playwright or, um, React's, uh, testing library.

Any of those, um, you- you've definitely used tools like this. You can just give them back to the model, uh, uh, to your agent, and they can run them. Give it to them as a tool, and they can, uh, prove it out themselves, and you've made this reproducible bit of evidence that then you can use in this one. Next time you use it ... Next time that page changes, it can run it again and verify that nothing broke. It acts as a regression, um, and keeps everyone on the team moving forward.

One caution, word of caution, though, a s- uh, like a successful screenshot isn't proof. Um, make sure that it's actually showing, you know, the state that you expect it to be in, that it actually like ran the commands to get to that state, uh, to actually be in there, because otherwise it might just be like another lie. Uh, but you're seeing a screenshot on the, the PR and thinking, "Oh, yeah, this is, this is correct." But it might be causing different side effects. It might have taken a different path that's not actually possible within the UI. Um, so you wanna make sure that it actually like

runs start to finish, um, to check that.

Uh, yeah, and so once you build these loops that are, you know, durable, reliable, repeatable, uniform, the value of what you get or the benef- the main benefit is that a, a failure starts to drive future work, right? So if in one loop the agent fails, uh, you're busy talking to somebody at work, or you're starting yet another task, that agent can determine on its own, this is not quite correct. Let me go take another stab at it. Let me now run the command, verify again. Okay, now we're good. If you s- think about what that means at scale, that a single PR or an a- a single lane of work, if you

will, and an agent, you know, self-heals, self-corrects, figures it out on its own, you're not coming back to that particular PR at the end of the day and, and then being like, "Oh, I, I still have to get this all the way through." And if you just multiply that out, right, that, that's, that's really where, um, again, we just say this is foundational for speed and velocity.

Yeah. And, uh, how many of you write, uh, good error messages? Show of hands.

Yeah, nobody. No, we already know. Sir, what's your error message? I guess I'll hand.

System error nine.

There we go. Yeah.

Solid.

Um, well, you should. Uh, I think they're more important than ever because, uh, when your a- your, uh, app throws an error, guess what can interpret and run with that error really well? An agent. So, um, if you make it so that your failure is explicit about why it failed and give as much detail as possible h- about how it got into that state, then, uh, your agent can just like self-heal from that. So it's, um, very easy, very worthwhile to, to do that.

Uh, yeah, and so this is a concept that, you know, Nick and I talk about. Like, it, it sounds like we're being, uh, in some ways we're trying to wrap determinism around LLMs. Uh, but we actually don't wanna do that.

Yeah.

And that's also something to the spirit of what the Amp team is talking about. Uh, the agents are in, in- intelligent, they're, they're capable. We want them to be creative, actually, because oftentimes if they have the right constraints, and then oftentimes constraints can be, can equal freedom, right, in a way, uh, then they will come up with creative solutions and actually solve what you said. That's why you give them the outcome. That's why, you know, Nick's project he's experimenting with is outcome-driven development. Um, and so it's not about over- being overly prescriptive

in how the agent achieves things. Um, that can actually lead to lower performance in some cases. It's more about giving it the map and saying, "You must get here. You'll know you're here when this command returns this."

Yeah. You don't necessarily have to prescribe how it gets there, though. And this is true, more and more true every single day with like the frontier models. Fable, Astra, Sol 6.1, uh, Opus 5.5, they're all really good at being, like you ... Being creative is like the way I, I describe that. But if you give them, this is where I wanna get to, and don't necessarily tell them the path that they need to take, they might wander and make sure that they catch more edge cases. They might figure something out or ask you a question that unlocks something. But

if you're just like, "You have to get here and then here and then here," they may not sway from that. And you want them to sway a little bit, but get to the right, right end goal. And like I said, this is something that we, we've only come around to really appreciating in the latest batch of frontier models-

Absolutely

... uh, where it's really becoming possible. So going forward, it's gonna just get better.

Yep.

So when it's going, um, you wanna get into like, you know, the typical TDD, like red, green refactor loop with your agent. Um, we've talked-- You've heard about loops, you know, at every one of these AIE events, and, um, it's no different here. But you can use that within the verification piece itself. Verify this, reproduce it, show me, uh, fix it, and then show me the proof of that. And how are you gonna show me the proof? That's just as important as the fix that's gonna go in, because that's what's gonna lead- leave you alone and allow you to trust the model to

actually do the work and, and then when it says, "Hey, I'm done," you know that it's done because it's proved it to you in some way.

Absolutely.

Yep. So in this example, uh, you can run snack doctor. Let's just go ahead and run that real quick.

Uh, p snack doctor.

And so this, uh, this ran, and it's showing us, um, you know, a bunch of checks that it did. So it's checking that we have Node, that Chromium's installed, uh, that our session exists, uh, that it's actually running, and that it's successfully built. So that's like what we're running as a quick doctor check, just a bunch of different things. So then we run the next command, snack join ada@example.test

Sorry, we don't type anymore.

Yeah.

So we don't know how to type.

Okay. Gave us the result back. Snack storage.

Storage is empty. It didn't actually put Ada in there.

So this is the point at which, in the real world, your agent, you've just gotten the value out of this entire system. 'Cause at this moment, your agent realizes, "I am not done, and I'm not gonna drop this work on the floor."

Yep. And if you reload it, it'll do the same thing one more time.

Boom. We have now given our, uh, app eyes. It can see this, it can run these, it can, uh, ma- it, it can make sense and change the world of the application, the Midnight Snack application, and see the results of that, and then be able to interpret what happened from there.

So you can try that. You can... If we up- like, apply the persistence fix to it, uh, and add honestCopy to it, then we can, um, run the proof. Let's run that.

And that kicked off a browser, and it failed. I didn't actually run the test, but if we had... Or run the fix. If we had run the fix, uh, it would show that.

Okay. One of the things that we're, we're thinking about a lot lately, and talking about, and starting to roll out into more applications is this, like, concept of receipts. Um, you know, you can, you can take like a, a ticket, like a, a help ticket system where it's, there's a certain performance the way that a human would answer a ticket and consider it done well as an excellent, you know, customer response or service. And then what does the, um, like preserving that in some way, for example, right? Uh, why do we wanna do that? There- there's a ton of reasons, and we'll, we'll get into some of them. But, uh, again, if you think about

if it, if we all, uh, or if it seems that in some way many, many folks are heading towards, you know, running RSI harnesses and self-evolving code bases, uh, everything that we do today, everything that your agent does today, every run all day long, that is very valuable data. It's very valuable data right now because you can feed it into LLMs and, and get fixes that you need. It's incredibly valuable data in the future as tooling improves, and it gets easier and easier to set up RSI loops. Because then you've got a data set of exactly how your system

performed that you can create, uh, you know, very comprehensive evals for and have agents hill climb on, which means loop and see their performance, and then continue going until they get better and better and better, right?

Yep. And at that point, they're able to pass on their own using that evidence from the previous runs.

So now, uh, if we fix that code, uh, it now persists. Uh, but it can also lie. So after we reload the page, it could say waitlisted. Um, but what the screenshot a- might actually say is, "You're in. Uh, your seat is confirmed." Those don't exactly match up, and so that might be, like, a failure right there because they don't exactly match. So this is kinda getting away from the static testing piece of it. Like, how do we prove that, um, it worked without having to be, like, very

cup- tightly coupled to the actual text on the screen, um, for a very contrived example. Um, and so if we just run, like, you know, p snack, ugh, judge.

Should run that, and it failed, um, because even though it worked, uh, and it says that we're in, um, Jev did not... Or the, the model did not see that. Um, and it's because the, the confirmation text breaks the contract. It's not exactly one-to-one. So how would we fix that without adding a lot of tight coupling to everything that we're doing? The answer is the System 1 models, um, which we'll get into here.

Yeah, absolutely. So, uh, just to introduce it for anyone that hasn't, uh, heard of it before, the, you know, concept of System 1, System 2 comes from psychology actually as like the, uh, a useful model to think about how the human brain works and how we think. So System 1 is super fast decisions. It's, it's, you know, autonomous. Like, uh, someone swerved at me while I'm driving, and I, and I just corrected without even thinking about it kind of thing, right? Some- I was about to lose my balance, and I, I corrected it. Um, and then System 2 is like, I need to figure out what I'm gonna do with my life, so I'm gonna go book an Airbnb and, like, think about it

for three days. Um, and so, you know, applied to the world of LLMs, now we've got Jev, and then immediately after we've got Clef from Cloudflare, you know, two days later. Um, that, that's even faster. And so there's... It's, it's a really exciting time to think about all the ways that we can combine these tools together to make the new primitives that we're gonna continue to see in evolving AI architectures.

Yeah. To give... Put it another way, that light right over there, if I said, "What color is that projecting?" Quickly you would say yellow. That's like System 1 thinking, right? You just... You know the color. If I said, "Why is it projecting yellow?" Well, bec-

Now we're thinking

... now we're thinking. You know, we could go down, like, a physics of why the light emits yellow.

Why are we here?

Yeah.

Like...

Uh, and that's how we got to really what we wanna talk to you about today.

Yeah.

Um, so in this, uh, we've got another game, uh, that we can play within, um, within this. It's a quick game with, um, the Snack. We have Saboteur and a Detective. Uh, and so, um, in this the, the Saboteur will pick a snack. Uh, it'll put a secret bug in there, uh, randomly putting a bug in there. Um, we don't know which. It'll just be random. And then you, the Detective, uh, find it with, uh, with Snack Alone, and then prove, uh, what that is. Name the bug and give a, give proof that you

found it. Uh, so this is something that you can do if you've pulled down the website, or pulled down the repo. Um, you can just run pnpm snack up --bug random, and, uh, give it... Uh, let it run and see if you can find it. So let's try that. We'll just give it some time to go.

Uh, what was it? --bug-

--bug random

Oh, it's already running. Let me kill that. Kill the right one.

Okay. So in this, uh, it ran. Uh, it's got the app URL. It's running. It's not headless, so we can see it. The bug, uh, says hidden until you stomp up. Yeah, right. Um, leave this running. In another terminal, run pnpm snack doctor. So let's run that.

All right. So that ran the whole thing, and we can see Node's there, Chrome's there. It's got a session. The proof here, uh, the last receipt says fail, but the source is now, uh, that proof is for code that no longer exists, so it wasn't able to prove that. That was a previous run. Let's run it again. Same thing. Uh, can anyone tell me what the bug is? Any ideas?

Wrong email?

The port's wrong.

Yep, exactly. Port's wrong.

Port 483.

Uh, I think it was running at 62174. No, that was right. Let me try it one more time. We got less than a minute.

Do you need to have a Jev API key for that one?

Uh, for that one-

Yeah

... yeah, you do.

The, the tools make it optional though, so you can run through all the exercises if you don't have it, and it'll just go. But, um, it's also a good opportunity to experiment with one if you do.

Yeah.

They're, they're pretty cheap calls.

Uh, they give you $5 for free when you sign up, and I think that'll last you, like, three years.

Yeah.

All right.

So, um, yeah, go ahead, Nick.

Okay.

Yeah. Thanks.

Um, so yeah, we ran it a couple times. I think there was some problem with my output. I'll debug that. Uh, it's funny, I need some verification tests around my verificat- my contrived verification examples. Um, but the, you know, the, the outcomes that could come up are, uh, that it named the bug, uh, with output, uh, w- said it works, but it didn't, uh, said it works, but it did, uh, blamed a bug that wasn't there, um, and then had

no ... It was the right bug or no evidence.

Right. And this is just getting back to that point of, uh, a contrived example, of course, but your team might score differently or have different criteria that you're looking at. This is super flexible. The most important thing to do is to start and to create one, create an eval baseline and to use it consistently and to evolve it together as a team. And then get to the point where your changes and your PRs automatically flex that eval so that you can get to the point where you know if changes are safe to land or if they're improving things or if they're making things worse or if they're not changing things at all.

Yep.

Uh, and every time it runs and it misses, um, that's a new line that can go in the map. So it can give us ideas of, like, the steps that we're getting there, why we're getting there, and kind of like, you know, every time it fails, it's getting us one step closer 'cause we can fix that. So the first time, uh, is memory only. Um, like it's not storing anywhere but in memory. Uh, we're putting in a wrong email address. Uh, we've lost our confirmation. Every time we can prove that we've caught that and then we don't run into that again.

Um, but then the overpromise one, uh, the screenshot or the judge running, uh, can tell us that it, um, it actually did pass when it didn't.

Same thing with console error. Uh, there might not be anything in the console, um, but it's in the receipt, or it might not be ... It might be getting caught, um, before it gets to the console. Um, yeah, anything like that. Or it could just be none. There's nothing to find. Everything works, um, and that's a real answer as well.

Um, yeah, if you run the doctor, kind of like what we were doing, it was showing this where it was showing the previous, uh, examples. Um, it's showing proof that no longer exists. Uh, that's because your code might have changed quite a bit between yesterday and today. Um, it might be different circumstances like, uh, a famous date bug, um, for example. It might be a, a number of different things. So, uh, we wanna build the tools in such a way that they're flexible enough to, uh, not get caught by, um,

issues like this. Uh, this is all kind of like going back to, you know, basic unit testing examples, uh, but bringing that kind of like deterministic check to the models in a way that they can self-heal and self-improve from there.

Absolutely.

Okay. So th- uh, this again to tie it back to what we used to do as human developers, right? What we used to be able to see, how we would flex the application, how we would check everything. Uh, that's really what we've been talking about and moving it into code. So by moving it into code and making it, uh, available in a deterministic CLI, then we can get to the same path and the same experience that we would flex as, you know, developers when ... in Chrome trying to fix something late at night. And that's really what unlocks the, the speed and the evolution and the ability to scale up the amount of code that you're delivering, and

to do that in a more safe way.

Uh, again, you know, if you, uh, the recommendation is if you have a web app, uh, and you're not already... You don't already have a custom verifier CLI, that's one of the, the highest leverage things you could start with for your, uh, code base. The foundational eval as well, that shows what good looks like in your application, and then also giving your agents the ability to see all aspects of performance, and, uh, networking, and, and traffic, and, and rendering, et cetera. And so giving it Chrome DevTools if you're in, um, if you're in the web app land. If you're

shipping native clients or shipping iOS, you know, or OSX apps, then there's an alternative, uh, projects and more coming out every day that you're then going to attach to there. But the, the, the common, you know, foundation here across all of them is that you need to give your agent the ability to see absolutely everything that you would normally see as a developer.

Yeah. The more that you can give it, the more arms or eyes that you can give it, the more it's going to be able to take over from you. Uh, and then you just have to give it the intuition about when to use those, how to use those. Think about the way that you decide, okay, if I... This bug came in, how would I reproduce this? Think about that, and if you can break it down, and this is dependent on your application and the type of, of code that you're writing, um, you can... You might be able to do that in a way where it can like think, oh, you know, for us it's a lot of sign-on things. Like, is it properly

setting a cookie? Is it setting an expiration in that? Like all of that.

The claims in the, in the job, right?

The correct claims. Yeah.

You can go... A- again, like the... This is applicable to every, you know, f- field and industry that you've got software for. Um, but it's, it's really just all about like, hey, that world where it made sense for us all to hand chisel code all the time is at least evolving or at least changing rapidly. What are the foundational... Like what's the, what are the same system fundamentals that we bring to this now so that we can accelerate software development?

Yep.

Uh, so yeah, doing expensive call stacks. Uh, but the, the important thing is that, uh, getting that output, making sure that it's giving you detailed output about what happened, uh, the state of it, uh, all of that so that it can be audited by the application, uh, by the model, and then you can run it again. It can fix it, run it again, and verify that those changes actually took hold and that there's no longer an issue.

Uh, yeah. And again, uh, can't overemphasize the value of keeping receipts. So if you have runs, uh, you know, it can be the entire user journey. Let's say it's an e-commerce app. It's like when they log in, they hit the page. What happened? What was their flow? Did they see an error at all? Did they see seven seconds of time to see your catalog, you know, come into view? Uh, were they able to put something in their cart? Was there cart abandonment? Um, and so tr- save that all now because every single one of those runs and receipts can be fed to LLMs. Or you can even set up a system that's running all night

long and is looking for, uh, the latest receipts being produced and is aggregating them and saying, "Well, today we saw eight failures in the caching module, so I'm obviously gonna, you know, issue a PR tonight to fix that properly." And then, oh, I have the tools I need to prove in this PR that the issue I was trying to solve has been fixed. Uh, and then the recommendations that we're seeing from, from folks too that are achieving like super high scale PR, uh, shipping without errors, they're, they're also able to, you know, add, uh, the stuff that is

for us as humans in our oversight still. So not just screenshots of the feature, but like a recording of I walked through the app this way, this was my flow, this is what I experienced, these were the states, and I got to the goal. So that if I'm waking up and trying to rip through, you know, 15 important PRs in the, the first thing in the morning, I can first start watching that and make sure, okay, it actually got to the end and worked. Then I can p- you know, start looking at the code, uh, if it's something that I'm actually going to like human review. And, uh, again, all of that I think is going to be immensely valuable as we continue

to see, uh, evolve, self-evolving tooling really.

Hmm.

And, you know, think about what that means though, because when we get to the point where there's, uh, self-evolving tooling is just like the default and people are using it, uh, there will be a specific advantage for folks that have preserved, uh, clean data sets of things that they can say, "This is what we are trying to get to in this application, and we want you to get all the way there and rapidly improve it," right? If you're not sitting on a mountain of that and your competitors are and they have similar apps, they're gonna be able to go significantly faster in their development cycle.

Yep. And this is just building up that trust so that we can continue to improve upon our app as we go. Uh, but we've have these checks for everything that came before it. So we've got unit tests, we've got evals, we've got smoke tests, we've got whatever our, our application needs, and we've got a way for the model to be able to interpret and run those and improve them, uh, in such a way that if we add new features in, we can run all of the previous ones and make sure that we didn't actually break anything, um, no regressions, and we can just mount and climb that

evidence ladder, building evidence as we go, um, and ensure that our app is working, uh, exactly as expected. And that really becomes the, the spec that shows what our app does.

Okay, cool. So we've got, uh, just over five minutes. Wanna pause and see if there's, uh, additional questions. Yes, sir.

Uh, so does the skill in your repo enforce the model to run the checks regularly?

I, I would personally, um, I, I'm curious what, hear what you think. But I would, I would personally almost lean away from skills, 'cause w- in my experience it's, it's, you know, uh, up to the model whether or not it's gonna actually follow that path and use it, um, versus like having the CI actually enforce, uh, that these, these commands get run as well.

Yeah.

As you're, uh, developing locally and you're, like if your, your agent requires you have a way to post call, post or something to say for all the time.

Yeah.

Yeah. You-

Absolutely.

Uh, like the answer or the question is how do you run these, th- these checks normally, right?

Yeah.

Effectively.

Yeah, I guess it's like, um, like I've had to run unit tests or regression tests or-

Yeah

... A tests and, um-

Yeah. It, it can, you can run those, you can run these regularly. You can have them be, like, part of CI, for example. You can have them be, um, various hooks that could be, like, a, a Git hook, although that's probably the last one I would reach for. But you could have it be, like, a Claude code hook. It depends on what your team is using and how they're, they're doing it. I might build, like, a full, uh, Pi extension to keep things running in the background, and just kind of like feed in information as it's going. Um, there, there's a number of different ways that you can do that. Uh, but the easiest way by far to

get everyone across the board, and I can say this now, just put an AGENTS.md, not a CLAUDE.md, uh, in your, in your app, and then have it say like, you know, o- on change verify things by running this. Or like every time you change significant things, or before you open a PR, do this. Like, whatever wording that you wanna put in there, and then, uh, that's an easy way to get everyone across the board, and they really kind of have to go out of their way to, uh, ig- ignore that or avoid it.

Yeah. Hooks are also relevant too if you're doing the local development-

Yeah

... right, for the whole team, 'cause you can share hooks across the board.

So you're saying you're presenting this more as initially as a CI tool, and then, but you could also use it, um, as your regular local dev?

Yeah.

Absolutely. Yeah.

Uh, well, CI is a, is a, I think a feature of it, but it's not... Like, the, the actual tool is like the rapid development that you get locally. So I give my agent a task to do, and I can just quickly run, uh... Like, like let the agent go. You know, I basically tell it, "This is where I wanna end up. I want the wait list to be... I wanna be able to add an email to the wait list, and then it gets persisted, uh, and saved, and I can verify that. Uh, so this is how you verify it. It goes into this table," blah, blah, blah. Like, whatever it might be. Um-

One second. Sorry.

Yeah.

I'm just trying to understand.

No, good.

And then I have a quick follow-up question.

Sure.

Um, with-- I imagine that before you all wrote this, you had, uh, like, Playwright end-to-end.

Yeah.

And, and end-to-end tests. What were the big, uh, benefits that your team got out of having those in CI and having your agent locally regularly run those versus, uh, this approach?

Yeah. What's the benefit of running it in CI versus like a local-

What benefits did you see? Not

Sure.

Yeah.

Uh, the benefits are definitely, like, a faster feedback loop, which allows us to run faster, and allows me to, uh, parallelize myself, because I'm no longer working on just one thing at a time. I'm working on at least... Right now I have three agents running. Uh, that's a very low day for me, but I'm at a conference. Um . Normally it's like 10 to 15.

He's really taking it easy today.

Um, and, and the, the whole idea is like I'm going to set it off, and I want... I don't want it to be waiting on me and saying, "Yep, I did it." And then I go run it and it's like, "No, that's not actually correct." And then I have to like set it off again, and I go get distracted by another agent again, and I just continue that loop. I'm just, like, constantly moving between projects. So I wanna give it the tools to where it can prove to itself that it did it, and then prove to me that... Like, show me the evidence in a way that I can trust, uh, so that when I come back to it, it might be in five minutes, it might be in an hour or a

couple of hours, uh, I wanna know that it has done the best possible job that it can to get done, to a state of done so that I'm not just like re-prompting it and starting that whole loop again. Because I don't wanna be... That's, that's not the fun part of my job anymore.

Yeah. I would, I would say another way to, to put it is that, and we haven't every day achieved this perfectly, but I want to have all of the different lanes that I started that day for stuff that I felt was important to ship that day, I wanna come back to them at the end and see them basic- like, correctly done.

Yeah.

What I don't wanna do is have a whole long day, and have them running in the background, and be doing other stuff, and going to meetings, and then come back, and then actually have to turn by turn nudge any of them.

Yeah.

Because that is, like, a significant slowdown, and it, it prevents me from going faster.

And another way to put this is I'm... Like, I rely on CI and the tests that are running in CI, and the code reviews. Like, we... How many people do automated code reviews with like Greptile, CodeRabbit, Codo, any of those? Yeah.

Yep.

I rely on those. We use Greptile. Greptile gives me a X out of five score, and if it gives me a five out of five and all of my CI is passing, I feel extremely confident that I'm not going to break production when this gets shipped. Uh, we've gotten to that level of trust with Greptile.

Yeah.

The more important thing for this is that I get to that state faster, and that, um-

With less turns

... with less turns, yes. And that, you know, I, I might add a whole... Like, I could add 10,000 lines of absolutely nothing, and it does nothing, and great, it passed. Greptile gave it a five out of five for some terrible reason. And then it goes into production, and it doesn't do anything, it doesn't break anything. Now it's just an extra 10,000 lines of nothing. Um, and so this is also to help me make sure that, like, I'm, I'm telling the agent, like, exactly what I want, and it gets there without going through a bunch of, um, additional code

to... Or additional like... Like, it stays on track, uh, but it also knows how to fix itself to keep itself on track while I'm off doing something else.

Okay. I might have time for one more quick question, but I think we're... Yes. Yes, sir.

Have you incorporated these processes in, uh, uh, in the production code that you're shipping to the customer, not just-

Yes

... locally?

Yes.

And how confident do you feel about the entire, uh, like second layer in, uh, your, uh, uh, Greptile? It, uh, gives a go ahead, but will it be how you monitor it before-

Yep

... and then you just test it once in production and-

You could also, you could also do-

Okay

... depending on what your stack, like there's, it's also, you know, there's various providers that you could easily also go to preview first and me- make... And then verify it there in a different way. So we'll then go to prod. And then also immediately monitor. It's another pattern we're starting to see emerge, is that when folks have a deployment that was mostly agentically implemented, there's also at least one agent specifically concerned with that production rollout looking for, watching all the metrics, watching Datadog, watching feature flags, right? Being able to say, "This is a- this is actually bad, and we're, we're reverting it."

Um, so I, I think that, like... I, I truly understand and get the, like, queasiness around it, like we all have are dealing with that, but I, I, I do see a path for that, um, because it just seems like, it seems inevitable. Yeah. Yep.

And that's all the time we have, but if you have more questions, uh, we'd be happy to chat. We'll be in the, uh, expo hall, booth six, right, uh, to the right when you come in. Uh, come find us. We'd be happy to cha- uh, to chat, show you what we're working on, uh, and talk to you about, uh, anything you'd like.

Yeah. Thank you so much-

Thank you

... for coming and your attention and everything. Appreciate it.