# Your Judge Is Hallucinating Too Source: https://www.youtube.com/watch?v=ZZ19zhYnDKY Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session. [06:11:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22303s) >> All right. Good afternoon, everybody. I [06:11:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22306s) am Brian Kennedy. I'm an engineer on [06:11:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22309s) Zoom Info's Agentic platform. [06:11:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22312s) Now, I want to start with showing you [06:11:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22315s) something one of our agents wrote. This [06:11:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22318s) is a real output from our product. The [06:12:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22320s) agent's job was to schedule an email for [06:12:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22323s) a customer. And this is what it reported [06:12:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22326s) back. It [clears throat] said it [06:12:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22328s) scheduled the email. It queued it up and [06:12:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22329s) that it was going to go out on Tuesday. [06:12:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22332s) We don't just trust the agent, though. A [06:12:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22334s) second model checks the agent's work. [06:12:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22336s) I'm going to call this one the judge. [06:12:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22338s) The judge read this and it accepted it. [06:12:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22341s) Great. [06:12:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22343s) Now, here's the problem. And this is [06:12:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22345s) what it actually happened there. Uh, [06:12:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22348s) these are the tool results. The record [06:12:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22350s) of what actually ran when the agent [06:12:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22352s) called the tools. I'm going to call [06:12:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22353s) these the receipts. The agent tried to [06:12:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22356s) create a record in Salesforce and it [06:12:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22357s) failed. The error message that came back [06:12:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22359s) says the record was not created and it [06:12:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22362s) tells the agent not to report a success [06:12:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22364s) or even make up a record. We wrote this [06:12:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22367s) message ourselves for exactly this [06:12:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22369s) situation. The agent tried again. It [06:12:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22372s) created a task which is a to-do. [06:12:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22375s) No email was ever scheduled. The agent [06:12:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22378s) doesn't even have a tool that can send [06:12:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22379s) an email. The agent made it up and the [06:13:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22381s) judge approved it. So that's two [06:13:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22384s) failures. The agent fabricated a send [06:13:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22387s) and the judge rubber stamped it, [06:13:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22389s) approved uh a claim that never actually [06:13:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22392s) checked. So if you build agents, you [06:13:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22394s) probably have run into this and you [06:13:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22396s) probably have a judge like this. It [06:13:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22397s) might be a code reviewer or a greater of [06:13:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22399s) your evals or a supervisor in your loop. [06:13:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22402s) Anytime a model decides whether another [06:13:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22405s) model's work is good, this can happen. [06:13:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22408s) But for the next 10 minutes, I'll show [06:13:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22410s) you how we caught this and what we did [06:13:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22412s) about it. First, some quick background. [06:13:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22415s) My team at Zoom Info um is called Agent [06:13:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22419s) Teams. We announced this last week. Uh [06:13:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22421s) it's a multi- aent orchestration layer [06:13:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22423s) that lives in Zoom Info's GTM studio. It [06:13:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22426s) coordinates a team of specialized agents [06:13:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22428s) across a company's systems to run GTM [06:13:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22431s) workflows. For example, find companies, [06:13:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22434s) research them, write an email, log it [06:13:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22436s) into their CRM. Each agent has its own [06:13:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22438s) roles and instructions like and tools [06:14:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22441s) like Salesforce, HubSpot, Slack. These [06:14:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22444s) agents make real uh decisions and take [06:14:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22447s) real actions in a customer's systems. [06:14:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22449s) They write records and they send [06:14:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22450s) messages. [06:14:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22453s) Every team and every agent finishes a [06:14:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22455s) step. A second model checks its work. [06:14:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22458s) That's the judge. And it has three [06:14:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22460s) options. To move on, to redo the step, [06:14:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22464s) or to stop the job altogether. You've [06:14:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22466s) probably heard this called LLM as a [06:14:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22468s) judge. It's the same idea as a code [06:14:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22470s) reviewer or an eval. [06:14:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22473s) Now, the hard part for us is that it's [06:14:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22476s) one model and it has to grade every kind [06:14:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22478s) of agent that we're running through it. [06:14:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22480s) Our customers build their own workflows [06:14:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22482s) with any agent and any tool in any [06:14:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22484s) order. So the permutations are almost [06:14:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22486s) infinite. So the list of jobs for the [06:14:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22489s) judge has to grow with it. The judge can [06:14:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22492s) tell whether the work was looks [06:14:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22494s) finished, but what it can't tell is [06:14:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22496s) whether the work was right because that [06:14:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22498s) depends on what right means for each [06:15:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22500s) job. And we're asking it the same [06:15:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22503s) question for all of them. [06:15:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22506s) So here's where we're going. First, I'll [06:15:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22509s) talk about what we found when we checked [06:15:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22510s) the judge. [06:15:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22513s) Then what we chose instead and what you [06:15:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22516s) can to do today with your own agents. [06:15:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22521s) To start, we ran offline evals. We took [06:15:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22524s) 500 real steps from production, each [06:15:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22527s) with a decision that the judge had made. [06:15:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22530s) Then we took a larger frontier model and [06:15:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22531s) regraded every one of those to come back [06:15:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22534s) with an accept or a send back. Then we [06:15:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22537s) hand checked each of those samples and [06:15:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22539s) we checked the grades where we disagreed [06:15:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22541s) with the model. We went with our call. [06:15:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22544s) This gave us a reference label for all [06:15:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22546s) 500 steps. [06:15:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22549s) And it turned out the judge was right [06:15:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22551s) 65% of the time. So more or less a coin [06:15:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22554s) flip. It accepted 118 steps that it [06:15:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22558s) should have sent back. That's more than [06:16:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22560s) one in four of everything that it [06:16:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22562s) accepted. [06:16:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22564s) Pretty bad. And when we read through [06:16:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22566s) those, the most common problem was that [06:16:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22569s) similar to the email at the start, the [06:16:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22572s) agent reported that it did something [06:16:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22574s) like send a message and no tool call had [06:16:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22577s) ever done that. [06:16:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22580s) So to see why the judge had accepted it, [06:16:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22582s) let's go back to the email and look at [06:16:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22584s) what the judge was given. On the left, [06:16:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22586s) you have what the agent's own summary. [06:16:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22589s) Think of it as like a story. It's [06:16:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22591s) confident. It's well written. It says [06:16:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22593s) the email was scheduled. Now on the [06:16:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22595s) right you have the receipts. This is the [06:16:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22597s) actual tool results. There are errors [06:16:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22600s) failed to create messages uh telling the [06:16:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22603s) agent not to make up the record. Now the [06:16:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22606s) judge had both of these and it went with [06:16:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22608s) the story. The work looked fine on the [06:16:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22610s) surface so it accepted it. [06:16:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22614s) Now this isn't only our judge. You can [06:16:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22616s) think of a code review agent that reads [06:16:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22619s) the PR description before it looks at [06:17:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22621s) the diff. It's a similar kind of [06:17:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22622s) problem. If the description says, "I [06:17:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22624s) fixed the bug, added tests," the agent [06:17:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22627s) is going to lean on that when it reads [06:17:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22628s) the code. [06:17:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22630s) So, the obvious fix at the start was to [06:17:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22632s) tune the judge and write a better [06:17:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22634s) prompt. For us, that was going to be a [06:17:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22636s) significant amount of work. Our agents [06:17:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22638s) write sales emails, update CRM, do [06:17:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22640s) research, and to judge each of these. [06:17:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22643s) Well, we'd have to figure out what it [06:17:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22645s) was that the user was asking. Give the [06:17:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22647s) judge the right rules for every kind of [06:17:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22650s) work and keep it all up to date. That's [06:17:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22653s) basically a custom judge for every use [06:17:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22654s) case. [06:17:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22656s) What we went with, we went a different [06:17:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22657s) direction. We stopped asking one big [06:17:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22660s) general question. Is this good? And we [06:17:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22662s) started asking a short list of small yes [06:17:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22665s) or no questions. They are the same [06:17:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22668s) questions for every step. And each one [06:17:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22670s) is small enough to answer by looking at [06:17:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22672s) the receipts. [06:17:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22674s) The first option here is basically a [06:17:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22676s) prompt tuning project that never ends. [06:17:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22678s) Whereas the second one is a small [06:18:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22680s) handful of small checks. [06:18:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22683s) Now to answer those questions, we use [06:18:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22686s) Jev. You've heard a lot about Jev today, [06:18:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22687s) so I'll skip that and keep that short. [06:18:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22690s) It's a tight decision model, not a [06:18:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22692s) generative model. Uh for mom at home [06:18:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22694s) watching, just ask Claude. [06:18:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22696s) and we'll send it structured facts, you [06:18:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22698s) know, about the step and a listed a type [06:18:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22701s) questions. A typed question is a yes or [06:18:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22704s) no question or a choice with from a [06:18:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22706s) fixed list of options. It sends back a [06:18:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22709s) probability for each answer and it [06:18:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22711s) answers all the questions in one call. [06:18:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22716s) We gave it four things. the [06:18:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22717s) instructions, the uh input, the receipts [06:18:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22721s) and meaning every tool call and whether [06:18:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22723s) it was successful or it failed and the [06:18:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22725s) agents report. Then we asked for one [06:18:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22728s) choice which was accept, redo or stop [06:18:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22732s) and that choice makes the decision. But [06:18:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22735s) then we also ask seven yes or no [06:18:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22737s) questions. Those seven questions tell us [06:19:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22740s) the why. They decide between a redo and [06:19:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22743s) a stop and they go into the feedback [06:19:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22746s) that the agent gets when it actually has [06:19:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22748s) to redo a step. [06:19:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22751s) Now back to the email. The LLM judge [06:19:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22754s) accepted it and it took about 2 and a [06:19:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22756s) half seconds. Jev gave us a probability [06:19:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22759s) for each option. For this step, it [06:19:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22762s) accepted 0%. Redo 97% and stop at 3%. [06:19:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22768s) And it took about a fifth of a second. [06:19:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22771s) We make the decision from the [06:19:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22773s) probability of accept. At 30% or higher, [06:19:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22776s) we accept that. At 5% or lower, we redo [06:19:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22780s) or stop. And anything in between means [06:19:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22782s) Jev isn't sure. So, we kick it to an LLM [06:19:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22784s) judge. We picked those cuto offs from [06:19:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22787s) testing the 500 labels that we had at [06:19:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22789s) the beginning. [06:19:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22792s) Now, to test whether what whether or not [06:19:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22794s) this works better, we took two different [06:19:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22796s) routes. The first test used the same 500 [06:19:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22799s) production steps from the offline eval [06:20:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22803s) was confident enough in about seven out [06:20:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22805s) of 10 of them. One of those steps uh the [06:20:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22808s) LM judge had accepted 82 that should [06:20:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22811s) have been sent back. Each red square [06:20:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22813s) here is one of those. Jev successfully [06:20:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22817s) caught 74 of those. The blue squares are [06:20:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22819s) the ones that Jeb would have sent back. [06:20:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22825s) Now the second test is what we call [06:20:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22827s) shadow mode. Both judges look at every [06:20:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22830s) step but only the LLM judge uh that [06:20:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22833s) decision counts. So we saved Jev what [06:20:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22836s) would have done and we don't let it [06:20:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22838s) affect the actual run. That gave us [06:20:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22840s) 37,000 decisions to mine for insights. [06:20:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22843s) And in that time there were 728 steps [06:20:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22847s) where a tool had clearly failed. The LLM [06:20:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22851s) caught 294 of them. Jev caught 623. [06:20:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22856s) And the fake email from the beginning, [06:20:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22858s) it wasn't a one-off thing either. The [06:21:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22860s) same step ran 1,081 times. And almost [06:21:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22863s) every time the agent said it had [06:21:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22865s) scheduled an email, the LLM judge only [06:21:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22867s) caught it once. Jev caught that 876 [06:21:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22870s) times. [06:21:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22874s) And when you have a judge, every step [06:21:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22876s) waits. Jev was 10 times faster and 40 [06:21:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22879s) times cheaper. At this scale, it is [06:21:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22882s) massive. [06:21:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22885s) [snorts] [06:21:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22896s) or stop. But next we're adding post [06:21:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22898s) condition checks which are questions [06:21:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22900s) that depend on which tool called uh was [06:21:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22903s) within the step. So we don't have to [06:21:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22905s) guess. Uh we know the step up front [06:21:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22907s) because the tool tells us. If a step [06:21:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22910s) called let's say a send tool, we also [06:21:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22912s) ask whether the message was delivered. [06:21:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22914s) If it wrote a record, we ask whether the [06:21:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22917s) record exists. And those go into Jev [06:21:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22919s) with the same seven questions in the [06:22:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22921s) same call. [06:22:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22924s) Now, what can you do today with your [06:22:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22926s) agents? [06:22:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22928s) First, pick one judgment. Somewhere in [06:22:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22930s) your systems, there is an LLM deciding [06:22:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22932s) whether or not something is good. It [06:22:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22934s) might be reviewing code, grading an [06:22:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22936s) eval. Pick one of those and rewrite it [06:22:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22938s) as a few simple yes or no questions that [06:22:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22940s) can be answered from the evidence. [06:22:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22943s) Second is build it. This is the easy [06:22:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22945s) part. Honestly, we used Jev and ours was [06:22:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22947s) seven questions. Then the third, ship it [06:22:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22950s) in shadow mode. run it next to what you [06:22:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22953s) already have, but don't let it make the [06:22:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22954s) decisions yet. Just save it. Uh, and [06:22:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22957s) then you can analyze it later. [06:22:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22959s) Then read the cases and see where the [06:22:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22962s) two disagree. And don't overthink it. [06:22:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22964s) Just start small. [06:22:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22967s) We had trusted our judge until we [06:22:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22968s) measured it. Measure yours. Run a second [06:22:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22971s) check. Run it in shadow mode and see [06:22:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22973s) where it disagrees. You may be surprised [06:22:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22975s) with what you find. Thank you.