Machine-generated source transcript; recognition errors remain. Timestamps and subtitles use the selected source recording’s clock, not clip-relative time. Section links use cited source moments; sections without a verified moment return to the talk start.
6:11:43>> All right. Good afternoon, everybody. I
6:11:46am Brian Kennedy. I'm an engineer on
6:11:49Zoom Info's Agentic platform.
6:11:52Now, I want to start with showing you
6:11:55something one of our agents wrote. This
6:11:58is a real output from our product. The
6:12:00agent's job was to schedule an email for
6:12:03a customer. And this is what it reported
6:12:06back. It [clears throat] said it
6:12:08scheduled the email. It queued it up and
6:12:09that it was going to go out on Tuesday.
6:12:12We don't just trust the agent, though. A
6:12:14second model checks the agent's work.
6:12:16I'm going to call this one the judge.
6:12:18The judge read this and it accepted it.
6:12:21Great.
6:12:23Now, here's the problem. And this is
6:12:25what it actually happened there. Uh,
6:12:28these are the tool results. The record
6:12:30of what actually ran when the agent
6:12:32called the tools. I'm going to call
6:12:33these the receipts. The agent tried to
6:12:36create a record in Salesforce and it
6:12:37failed. The error message that came back
6:12:39says the record was not created and it
6:12:42tells the agent not to report a success
6:12:44or even make up a record. We wrote this
6:12:47message ourselves for exactly this
6:12:49situation. The agent tried again. It
6:12:52created a task which is a to-do.
6:12:55No email was ever scheduled. The agent
6:12:58doesn't even have a tool that can send
6:12:59an email. The agent made it up and the
6:13:01judge approved it. So that's two
6:13:04failures. The agent fabricated a send
6:13:07and the judge rubber stamped it,
6:13:09approved uh a claim that never actually
6:13:12checked. So if you build agents, you
6:13:14probably have run into this and you
6:13:16probably have a judge like this. It
6:13:17might be a code reviewer or a greater of
6:13:19your evals or a supervisor in your loop.
6:13:22Anytime a model decides whether another
6:13:25model's work is good, this can happen.
6:13:28But for the next 10 minutes, I'll show
6:13:30you how we caught this and what we did
6:13:32about it. First, some quick background.
6:13:35My team at Zoom Info um is called Agent
6:13:39Teams. We announced this last week. Uh
6:13:41it's a multi- aent orchestration layer
6:13:43that lives in Zoom Info's GTM studio. It
6:13:46coordinates a team of specialized agents
6:13:48across a company's systems to run GTM
6:13:51workflows. For example, find companies,
6:13:54research them, write an email, log it
6:13:56into their CRM. Each agent has its own
6:13:58roles and instructions like and tools
6:14:01like Salesforce, HubSpot, Slack. These
6:14:04agents make real uh decisions and take
6:14:07real actions in a customer's systems.
6:14:09They write records and they send
6:14:10messages.
6:14:13Every team and every agent finishes a
6:14:15step. A second model checks its work.
6:14:18That's the judge. And it has three
6:14:20options. To move on, to redo the step,
6:14:24or to stop the job altogether. You've
6:14:26probably heard this called LLM as a
6:14:28judge. It's the same idea as a code
6:14:30reviewer or an eval.
6:14:33Now, the hard part for us is that it's
6:14:36one model and it has to grade every kind
6:14:38of agent that we're running through it.
6:14:40Our customers build their own workflows
6:14:42with any agent and any tool in any
6:14:44order. So the permutations are almost
6:14:46infinite. So the list of jobs for the
6:14:49judge has to grow with it. The judge can
6:14:52tell whether the work was looks
6:14:54finished, but what it can't tell is
6:14:56whether the work was right because that
6:14:58depends on what right means for each
6:15:00job. And we're asking it the same
6:15:03question for all of them.
6:15:06So here's where we're going. First, I'll
6:15:09talk about what we found when we checked
6:15:10the judge.
6:15:13Then what we chose instead and what you
6:15:16can to do today with your own agents.
6:15:21To start, we ran offline evals. We took
6:15:24500 real steps from production, each
6:15:27with a decision that the judge had made.
6:15:30Then we took a larger frontier model and
6:15:31regraded every one of those to come back
6:15:34with an accept or a send back. Then we
6:15:37hand checked each of those samples and
6:15:39we checked the grades where we disagreed
6:15:41with the model. We went with our call.
6:15:44This gave us a reference label for all
6:15:46500 steps.
6:15:49And it turned out the judge was right
6:15:5165% of the time. So more or less a coin
6:15:54flip. It accepted 118 steps that it
6:15:58should have sent back. That's more than
6:16:00one in four of everything that it
6:16:02accepted.
6:16:04Pretty bad. And when we read through
6:16:06those, the most common problem was that
6:16:09similar to the email at the start, the
6:16:12agent reported that it did something
6:16:14like send a message and no tool call had
6:16:17ever done that.
6:16:20So to see why the judge had accepted it,
6:16:22let's go back to the email and look at
6:16:24what the judge was given. On the left,
6:16:26you have what the agent's own summary.
6:16:29Think of it as like a story. It's
6:16:31confident. It's well written. It says
6:16:33the email was scheduled. Now on the
6:16:35right you have the receipts. This is the
6:16:37actual tool results. There are errors
6:16:40failed to create messages uh telling the
6:16:43agent not to make up the record. Now the
6:16:46judge had both of these and it went with
6:16:48the story. The work looked fine on the
6:16:50surface so it accepted it.
6:16:54Now this isn't only our judge. You can
6:16:56think of a code review agent that reads
6:16:59the PR description before it looks at
6:17:01the diff. It's a similar kind of
6:17:02problem. If the description says, "I
6:17:04fixed the bug, added tests," the agent
6:17:07is going to lean on that when it reads
6:17:08the code.
6:17:10So, the obvious fix at the start was to
6:17:12tune the judge and write a better
6:17:14prompt. For us, that was going to be a
6:17:16significant amount of work. Our agents
6:17:18write sales emails, update CRM, do
6:17:20research, and to judge each of these.
6:17:23Well, we'd have to figure out what it
6:17:25was that the user was asking. Give the
6:17:27judge the right rules for every kind of
6:17:30work and keep it all up to date. That's
6:17:33basically a custom judge for every use
6:17:34case.
6:17:36What we went with, we went a different
6:17:37direction. We stopped asking one big
6:17:40general question. Is this good? And we
6:17:42started asking a short list of small yes
6:17:45or no questions. They are the same
6:17:48questions for every step. And each one
6:17:50is small enough to answer by looking at
6:17:52the receipts.
6:17:54The first option here is basically a
6:17:56prompt tuning project that never ends.
6:17:58Whereas the second one is a small
6:18:00handful of small checks.
6:18:03Now to answer those questions, we use
6:18:06Jev. You've heard a lot about Jev today,
6:18:07so I'll skip that and keep that short.
6:18:10It's a tight decision model, not a
6:18:12generative model. Uh for mom at home
6:18:14watching, just ask Claude.
6:18:16and we'll send it structured facts, you
6:18:18know, about the step and a listed a type
6:18:21questions. A typed question is a yes or
6:18:24no question or a choice with from a
6:18:26fixed list of options. It sends back a
6:18:29probability for each answer and it
6:18:31answers all the questions in one call.
6:18:36We gave it four things. the
6:18:37instructions, the uh input, the receipts
6:18:41and meaning every tool call and whether
6:18:43it was successful or it failed and the
6:18:45agents report. Then we asked for one
6:18:48choice which was accept, redo or stop
6:18:52and that choice makes the decision. But
6:18:55then we also ask seven yes or no
6:18:57questions. Those seven questions tell us
6:19:00the why. They decide between a redo and
6:19:03a stop and they go into the feedback
6:19:06that the agent gets when it actually has
6:19:08to redo a step.
6:19:11Now back to the email. The LLM judge
6:19:14accepted it and it took about 2 and a
6:19:16half seconds. Jev gave us a probability
6:19:19for each option. For this step, it
6:19:22accepted 0%. Redo 97% and stop at 3%.
6:19:28And it took about a fifth of a second.
6:19:31We make the decision from the
6:19:33probability of accept. At 30% or higher,
6:19:36we accept that. At 5% or lower, we redo
6:19:40or stop. And anything in between means
6:19:42Jev isn't sure. So, we kick it to an LLM
6:19:44judge. We picked those cuto offs from
6:19:47testing the 500 labels that we had at
6:19:49the beginning.
6:19:52Now, to test whether what whether or not
6:19:54this works better, we took two different
6:19:56routes. The first test used the same 500
6:19:59production steps from the offline eval
6:20:03was confident enough in about seven out
6:20:05of 10 of them. One of those steps uh the
6:20:08LM judge had accepted 82 that should
6:20:11have been sent back. Each red square
6:20:13here is one of those. Jev successfully
6:20:17caught 74 of those. The blue squares are
6:20:19the ones that Jeb would have sent back.
6:20:25Now the second test is what we call
6:20:27shadow mode. Both judges look at every
6:20:30step but only the LLM judge uh that
6:20:33decision counts. So we saved Jev what
6:20:36would have done and we don't let it
6:20:38affect the actual run. That gave us
6:20:4037,000 decisions to mine for insights.
6:20:43And in that time there were 728 steps
6:20:47where a tool had clearly failed. The LLM
6:20:51caught 294 of them. Jev caught 623.
6:20:56And the fake email from the beginning,
6:20:58it wasn't a one-off thing either. The
6:21:00same step ran 1,081 times. And almost
6:21:03every time the agent said it had
6:21:05scheduled an email, the LLM judge only
6:21:07caught it once. Jev caught that 876
6:21:10times.
6:21:14And when you have a judge, every step
6:21:16waits. Jev was 10 times faster and 40
6:21:19times cheaper. At this scale, it is
6:21:22massive.
6:21:25[snorts]
6:21:36or stop. But next we're adding post
6:21:38condition checks which are questions
6:21:40that depend on which tool called uh was
6:21:43within the step. So we don't have to
6:21:45guess. Uh we know the step up front
6:21:47because the tool tells us. If a step
6:21:50called let's say a send tool, we also
6:21:52ask whether the message was delivered.
6:21:54If it wrote a record, we ask whether the
6:21:57record exists. And those go into Jev
6:21:59with the same seven questions in the
6:22:01same call.
6:22:04Now, what can you do today with your
6:22:06agents?
6:22:08First, pick one judgment. Somewhere in
6:22:10your systems, there is an LLM deciding
6:22:12whether or not something is good. It
6:22:14might be reviewing code, grading an
6:22:16eval. Pick one of those and rewrite it
6:22:18as a few simple yes or no questions that
6:22:20can be answered from the evidence.
6:22:23Second is build it. This is the easy
6:22:25part. Honestly, we used Jev and ours was
6:22:27seven questions. Then the third, ship it
6:22:30in shadow mode. run it next to what you
6:22:33already have, but don't let it make the
6:22:34decisions yet. Just save it. Uh, and
6:22:37then you can analyze it later.
6:22:39Then read the cases and see where the
6:22:42two disagree. And don't overthink it.
6:22:44Just start small.
6:22:47We had trusted our judge until we
6:22:48measured it. Measure yours. Run a second
6:22:51check. Run it in shadow mode and see
6:22:53where it disagrees. You may be surprised
6:22:55with what you find. Thank you.
Original stream, 6:20:30. Both judges inspect the same steps; only the LLM judge controls the workflow.