1
06:11:43,840 --> 06:11:46,788
>> All right. Good afternoon, everybody. I

2
06:11:46,798 --> 06:11:49,190
am Brian Kennedy. I'm an engineer on

3
06:11:49,200 --> 06:11:52,470
Zoom Info's Agentic platform.

4
06:11:52,480 --> 06:11:55,430
Now, I want to start with showing you

5
06:11:55,440 --> 06:11:58,070
something one of our agents wrote. This

6
06:11:58,080 --> 06:12:00,150
is a real output from our product. The

7
06:12:00,160 --> 06:12:03,190
agent's job was to schedule an email for

8
06:12:03,200 --> 06:12:05,990
a customer. And this is what it reported

9
06:12:06,000 --> 06:12:08,070
back. It [clears throat] said it

10
06:12:08,080 --> 06:12:09,750
scheduled the email. It queued it up and

11
06:12:09,760 --> 06:12:12,150
that it was going to go out on Tuesday.

12
06:12:12,160 --> 06:12:14,470
We don't just trust the agent, though. A

13
06:12:14,480 --> 06:12:16,548
second model checks the agent's work.

14
06:12:16,558 --> 06:12:18,470
I'm going to call this one the judge.

15
06:12:18,480 --> 06:12:21,590
The judge read this and it accepted it.

16
06:12:21,600 --> 06:12:23,430
Great.

17
06:12:23,440 --> 06:12:25,910
Now, here's the problem. And this is

18
06:12:25,920 --> 06:12:28,948
what it actually happened there. Uh,

19
06:12:28,958 --> 06:12:30,628
these are the tool results. The record

20
06:12:30,638 --> 06:12:32,390
of what actually ran when the agent

21
06:12:32,400 --> 06:12:33,670
called the tools. I'm going to call

22
06:12:33,680 --> 06:12:36,070
these the receipts. The agent tried to

23
06:12:36,080 --> 06:12:37,590
create a record in Salesforce and it

24
06:12:37,600 --> 06:12:39,910
failed. The error message that came back

25
06:12:39,920 --> 06:12:42,390
says the record was not created and it

26
06:12:42,400 --> 06:12:44,708
tells the agent not to report a success

27
06:12:44,718 --> 06:12:47,190
or even make up a record. We wrote this

28
06:12:47,200 --> 06:12:49,030
message ourselves for exactly this

29
06:12:49,040 --> 06:12:52,150
situation. The agent tried again. It

30
06:12:52,160 --> 06:12:55,670
created a task which is a to-do.

31
06:12:55,680 --> 06:12:58,150
No email was ever scheduled. The agent

32
06:12:58,160 --> 06:12:59,670
doesn't even have a tool that can send

33
06:12:59,680 --> 06:13:01,670
an email. The agent made it up and the

34
06:13:01,680 --> 06:13:04,230
judge approved it. So that's two

35
06:13:04,240 --> 06:13:07,670
failures. The agent fabricated a send

36
06:13:07,680 --> 06:13:09,350
and the judge rubber stamped it,

37
06:13:09,360 --> 06:13:12,230
approved uh a claim that never actually

38
06:13:12,240 --> 06:13:14,708
checked. So if you build agents, you

39
06:13:14,718 --> 06:13:16,070
probably have run into this and you

40
06:13:16,080 --> 06:13:17,510
probably have a judge like this. It

41
06:13:17,520 --> 06:13:19,830
might be a code reviewer or a greater of

42
06:13:19,840 --> 06:13:22,390
your evals or a supervisor in your loop.

43
06:13:22,400 --> 06:13:25,190
Anytime a model decides whether another

44
06:13:25,200 --> 06:13:28,230
model's work is good, this can happen.

45
06:13:28,240 --> 06:13:30,230
But for the next 10 minutes, I'll show

46
06:13:30,240 --> 06:13:32,150
you how we caught this and what we did

47
06:13:32,160 --> 06:13:35,750
about it. First, some quick background.

48
06:13:35,760 --> 06:13:39,190
My team at Zoom Info um is called Agent

49
06:13:39,200 --> 06:13:41,510
Teams. We announced this last week. Uh

50
06:13:41,520 --> 06:13:43,590
it's a multi- aent orchestration layer

51
06:13:43,600 --> 06:13:46,628
that lives in Zoom Info's GTM studio. It

52
06:13:46,638 --> 06:13:48,868
coordinates a team of specialized agents

53
06:13:48,878 --> 06:13:51,350
across a company's systems to run GTM

54
06:13:51,360 --> 06:13:54,390
workflows. For example, find companies,

55
06:13:54,400 --> 06:13:56,470
research them, write an email, log it

56
06:13:56,480 --> 06:13:58,948
into their CRM. Each agent has its own

57
06:13:58,958 --> 06:14:01,350
roles and instructions like and tools

58
06:14:01,360 --> 06:14:04,310
like Salesforce, HubSpot, Slack. These

59
06:14:04,320 --> 06:14:07,110
agents make real uh decisions and take

60
06:14:07,120 --> 06:14:09,110
real actions in a customer's systems.

61
06:14:09,120 --> 06:14:10,708
They write records and they send

62
06:14:10,718 --> 06:14:13,110
messages.

63
06:14:13,120 --> 06:14:15,670
Every team and every agent finishes a

64
06:14:15,680 --> 06:14:18,628
step. A second model checks its work.

65
06:14:18,638 --> 06:14:20,868
That's the judge. And it has three

66
06:14:20,878 --> 06:14:24,150
options. To move on, to redo the step,

67
06:14:24,160 --> 06:14:26,788
or to stop the job altogether. You've

68
06:14:26,798 --> 06:14:28,230
probably heard this called LLM as a

69
06:14:28,240 --> 06:14:30,390
judge. It's the same idea as a code

70
06:14:30,400 --> 06:14:33,830
reviewer or an eval.

71
06:14:33,840 --> 06:14:36,150
Now, the hard part for us is that it's

72
06:14:36,160 --> 06:14:38,310
one model and it has to grade every kind

73
06:14:38,320 --> 06:14:40,310
of agent that we're running through it.

74
06:14:40,320 --> 06:14:42,470
Our customers build their own workflows

75
06:14:42,480 --> 06:14:44,628
with any agent and any tool in any

76
06:14:44,638 --> 06:14:46,548
order. So the permutations are almost

77
06:14:46,558 --> 06:14:49,110
infinite. So the list of jobs for the

78
06:14:49,120 --> 06:14:52,548
judge has to grow with it. The judge can

79
06:14:52,558 --> 06:14:54,470
tell whether the work was looks

80
06:14:54,480 --> 06:14:56,390
finished, but what it can't tell is

81
06:14:56,400 --> 06:14:58,628
whether the work was right because that

82
06:14:58,638 --> 06:15:00,868
depends on what right means for each

83
06:15:00,878 --> 06:15:03,190
job. And we're asking it the same

84
06:15:03,200 --> 06:15:06,868
question for all of them.

85
06:15:06,878 --> 06:15:09,430
So here's where we're going. First, I'll

86
06:15:09,440 --> 06:15:10,948
talk about what we found when we checked

87
06:15:10,958 --> 06:15:13,270
the judge.

88
06:15:13,280 --> 06:15:16,230
Then what we chose instead and what you

89
06:15:16,240 --> 06:15:21,430
can to do today with your own agents.

90
06:15:21,440 --> 06:15:24,310
To start, we ran offline evals. We took

91
06:15:24,320 --> 06:15:27,350
500 real steps from production, each

92
06:15:27,360 --> 06:15:29,990
with a decision that the judge had made.

93
06:15:30,000 --> 06:15:31,910
Then we took a larger frontier model and

94
06:15:31,920 --> 06:15:34,230
regraded every one of those to come back

95
06:15:34,240 --> 06:15:37,510
with an accept or a send back. Then we

96
06:15:37,520 --> 06:15:39,430
hand checked each of those samples and

97
06:15:39,440 --> 06:15:41,670
we checked the grades where we disagreed

98
06:15:41,680 --> 06:15:44,070
with the model. We went with our call.

99
06:15:44,080 --> 06:15:46,070
This gave us a reference label for all

100
06:15:46,080 --> 06:15:49,590
500 steps.

101
06:15:49,600 --> 06:15:51,910
And it turned out the judge was right

102
06:15:51,920 --> 06:15:54,310
65% of the time. So more or less a coin

103
06:15:54,320 --> 06:15:58,470
flip. It accepted 118 steps that it

104
06:15:58,480 --> 06:16:00,470
should have sent back. That's more than

105
06:16:00,480 --> 06:16:02,470
one in four of everything that it

106
06:16:02,480 --> 06:16:04,390
accepted.

107
06:16:04,400 --> 06:16:06,628
Pretty bad. And when we read through

108
06:16:06,638 --> 06:16:09,750
those, the most common problem was that

109
06:16:09,760 --> 06:16:11,990
similar to the email at the start, the

110
06:16:12,000 --> 06:16:14,150
agent reported that it did something

111
06:16:14,160 --> 06:16:17,110
like send a message and no tool call had

112
06:16:17,120 --> 06:16:20,070
ever done that.

113
06:16:20,080 --> 06:16:22,310
So to see why the judge had accepted it,

114
06:16:22,320 --> 06:16:24,310
let's go back to the email and look at

115
06:16:24,320 --> 06:16:26,868
what the judge was given. On the left,

116
06:16:26,878 --> 06:16:29,350
you have what the agent's own summary.

117
06:16:29,360 --> 06:16:31,270
Think of it as like a story. It's

118
06:16:31,280 --> 06:16:33,430
confident. It's well written. It says

119
06:16:33,440 --> 06:16:35,910
the email was scheduled. Now on the

120
06:16:35,920 --> 06:16:37,670
right you have the receipts. This is the

121
06:16:37,680 --> 06:16:40,708
actual tool results. There are errors

122
06:16:40,718 --> 06:16:43,590
failed to create messages uh telling the

123
06:16:43,600 --> 06:16:46,150
agent not to make up the record. Now the

124
06:16:46,160 --> 06:16:48,230
judge had both of these and it went with

125
06:16:48,240 --> 06:16:50,548
the story. The work looked fine on the

126
06:16:50,558 --> 06:16:54,788
surface so it accepted it.

127
06:16:54,798 --> 06:16:56,948
Now this isn't only our judge. You can

128
06:16:56,958 --> 06:16:59,190
think of a code review agent that reads

129
06:16:59,200 --> 06:17:01,110
the PR description before it looks at

130
06:17:01,120 --> 06:17:02,628
the diff. It's a similar kind of

131
06:17:02,638 --> 06:17:04,948
problem. If the description says, "I

132
06:17:04,958 --> 06:17:07,030
fixed the bug, added tests," the agent

133
06:17:07,040 --> 06:17:08,470
is going to lean on that when it reads

134
06:17:08,480 --> 06:17:10,470
the code.

135
06:17:10,480 --> 06:17:12,708
So, the obvious fix at the start was to

136
06:17:12,718 --> 06:17:14,708
tune the judge and write a better

137
06:17:14,718 --> 06:17:16,470
prompt. For us, that was going to be a

138
06:17:16,480 --> 06:17:18,470
significant amount of work. Our agents

139
06:17:18,480 --> 06:17:20,788
write sales emails, update CRM, do

140
06:17:20,798 --> 06:17:23,590
research, and to judge each of these.

141
06:17:23,600 --> 06:17:25,670
Well, we'd have to figure out what it

142
06:17:25,680 --> 06:17:27,830
was that the user was asking. Give the

143
06:17:27,840 --> 06:17:30,390
judge the right rules for every kind of

144
06:17:30,400 --> 06:17:33,030
work and keep it all up to date. That's

145
06:17:33,040 --> 06:17:34,868
basically a custom judge for every use

146
06:17:34,878 --> 06:17:36,390
case.

147
06:17:36,400 --> 06:17:37,910
What we went with, we went a different

148
06:17:37,920 --> 06:17:40,548
direction. We stopped asking one big

149
06:17:40,558 --> 06:17:42,948
general question. Is this good? And we

150
06:17:42,958 --> 06:17:45,430
started asking a short list of small yes

151
06:17:45,440 --> 06:17:48,150
or no questions. They are the same

152
06:17:48,160 --> 06:17:50,390
questions for every step. And each one

153
06:17:50,400 --> 06:17:52,310
is small enough to answer by looking at

154
06:17:52,320 --> 06:17:54,548
the receipts.

155
06:17:54,558 --> 06:17:56,548
The first option here is basically a

156
06:17:56,558 --> 06:17:58,868
prompt tuning project that never ends.

157
06:17:58,878 --> 06:18:00,470
Whereas the second one is a small

158
06:18:00,480 --> 06:18:03,910
handful of small checks.

159
06:18:03,920 --> 06:18:05,990
Now to answer those questions, we use

160
06:18:06,000 --> 06:18:07,910
Jev. You've heard a lot about Jev today,

161
06:18:07,920 --> 06:18:10,150
so I'll skip that and keep that short.

162
06:18:10,160 --> 06:18:11,990
It's a tight decision model, not a

163
06:18:12,000 --> 06:18:14,150
generative model. Uh for mom at home

164
06:18:14,160 --> 06:18:16,548
watching, just ask Claude.

165
06:18:16,558 --> 06:18:18,868
and we'll send it structured facts, you

166
06:18:18,878 --> 06:18:21,590
know, about the step and a listed a type

167
06:18:21,600 --> 06:18:24,070
questions. A typed question is a yes or

168
06:18:24,080 --> 06:18:26,548
no question or a choice with from a

169
06:18:26,558 --> 06:18:29,350
fixed list of options. It sends back a

170
06:18:29,360 --> 06:18:31,510
probability for each answer and it

171
06:18:31,520 --> 06:18:36,070
answers all the questions in one call.

172
06:18:36,080 --> 06:18:37,750
We gave it four things. the

173
06:18:37,760 --> 06:18:41,510
instructions, the uh input, the receipts

174
06:18:41,520 --> 06:18:43,190
and meaning every tool call and whether

175
06:18:43,200 --> 06:18:45,830
it was successful or it failed and the

176
06:18:45,840 --> 06:18:48,868
agents report. Then we asked for one

177
06:18:48,878 --> 06:18:52,708
choice which was accept, redo or stop

178
06:18:52,718 --> 06:18:55,750
and that choice makes the decision. But

179
06:18:55,760 --> 06:18:57,750
then we also ask seven yes or no

180
06:18:57,760 --> 06:19:00,070
questions. Those seven questions tell us

181
06:19:00,080 --> 06:19:03,350
the why. They decide between a redo and

182
06:19:03,360 --> 06:19:06,150
a stop and they go into the feedback

183
06:19:06,160 --> 06:19:08,230
that the agent gets when it actually has

184
06:19:08,240 --> 06:19:11,510
to redo a step.

185
06:19:11,520 --> 06:19:14,390
Now back to the email. The LLM judge

186
06:19:14,400 --> 06:19:16,470
accepted it and it took about 2 and a

187
06:19:16,480 --> 06:19:19,590
half seconds. Jev gave us a probability

188
06:19:19,600 --> 06:19:22,310
for each option. For this step, it

189
06:19:22,320 --> 06:19:28,230
accepted 0%. Redo 97% and stop at 3%.

190
06:19:28,240 --> 06:19:31,590
And it took about a fifth of a second.

191
06:19:31,600 --> 06:19:33,030
We make the decision from the

192
06:19:33,040 --> 06:19:36,948
probability of accept. At 30% or higher,

193
06:19:36,958 --> 06:19:40,070
we accept that. At 5% or lower, we redo

194
06:19:40,080 --> 06:19:42,708
or stop. And anything in between means

195
06:19:42,718 --> 06:19:44,868
Jev isn't sure. So, we kick it to an LLM

196
06:19:44,878 --> 06:19:47,430
judge. We picked those cuto offs from

197
06:19:47,440 --> 06:19:49,350
testing the 500 labels that we had at

198
06:19:49,360 --> 06:19:52,070
the beginning.

199
06:19:52,080 --> 06:19:54,310
Now, to test whether what whether or not

200
06:19:54,320 --> 06:19:56,390
this works better, we took two different

201
06:19:56,400 --> 06:19:59,910
routes. The first test used the same 500

202
06:19:59,920 --> 06:20:03,110
production steps from the offline eval

203
06:20:03,120 --> 06:20:05,190
was confident enough in about seven out

204
06:20:05,200 --> 06:20:08,470
of 10 of them. One of those steps uh the

205
06:20:08,480 --> 06:20:11,350
LM judge had accepted 82 that should

206
06:20:11,360 --> 06:20:13,830
have been sent back. Each red square

207
06:20:13,840 --> 06:20:17,270
here is one of those. Jev successfully

208
06:20:17,280 --> 06:20:19,910
caught 74 of those. The blue squares are

209
06:20:19,920 --> 06:20:25,670
the ones that Jeb would have sent back.

210
06:20:25,680 --> 06:20:27,270
Now the second test is what we call

211
06:20:27,280 --> 06:20:30,548
shadow mode. Both judges look at every

212
06:20:30,558 --> 06:20:33,510
step but only the LLM judge uh that

213
06:20:33,520 --> 06:20:36,070
decision counts. So we saved Jev what

214
06:20:36,080 --> 06:20:38,070
would have done and we don't let it

215
06:20:38,080 --> 06:20:40,548
affect the actual run. That gave us

216
06:20:40,558 --> 06:20:43,910
37,000 decisions to mine for insights.

217
06:20:43,920 --> 06:20:47,510
And in that time there were 728 steps

218
06:20:47,520 --> 06:20:51,110
where a tool had clearly failed. The LLM

219
06:20:51,120 --> 06:20:56,628
caught 294 of them. Jev caught 623.

220
06:20:56,638 --> 06:20:58,868
And the fake email from the beginning,

221
06:20:58,878 --> 06:21:00,788
it wasn't a one-off thing either. The

222
06:21:00,798 --> 06:21:03,750
same step ran 1,081 times. And almost

223
06:21:03,760 --> 06:21:05,350
every time the agent said it had

224
06:21:05,360 --> 06:21:07,510
scheduled an email, the LLM judge only

225
06:21:07,520 --> 06:21:10,470
caught it once. Jev caught that 876

226
06:21:10,480 --> 06:21:14,310
times.

227
06:21:14,320 --> 06:21:16,070
And when you have a judge, every step

228
06:21:16,080 --> 06:21:19,350
waits. Jev was 10 times faster and 40

229
06:21:19,360 --> 06:21:22,070
times cheaper. At this scale, it is

230
06:21:22,080 --> 06:21:25,298
massive.

231
06:21:25,308 --> 06:21:35,990
[snorts]

232
06:21:36,000 --> 06:21:38,788
or stop. But next we're adding post

233
06:21:38,798 --> 06:21:40,868
condition checks which are questions

234
06:21:40,878 --> 06:21:43,510
that depend on which tool called uh was

235
06:21:43,520 --> 06:21:45,670
within the step. So we don't have to

236
06:21:45,680 --> 06:21:47,750
guess. Uh we know the step up front

237
06:21:47,760 --> 06:21:50,470
because the tool tells us. If a step

238
06:21:50,480 --> 06:21:52,548
called let's say a send tool, we also

239
06:21:52,558 --> 06:21:54,868
ask whether the message was delivered.

240
06:21:54,878 --> 06:21:57,350
If it wrote a record, we ask whether the

241
06:21:57,360 --> 06:21:59,830
record exists. And those go into Jev

242
06:21:59,840 --> 06:22:01,270
with the same seven questions in the

243
06:22:01,280 --> 06:22:04,390
same call.

244
06:22:04,400 --> 06:22:06,310
Now, what can you do today with your

245
06:22:06,320 --> 06:22:08,788
agents?

246
06:22:08,798 --> 06:22:10,788
First, pick one judgment. Somewhere in

247
06:22:10,798 --> 06:22:12,708
your systems, there is an LLM deciding

248
06:22:12,718 --> 06:22:14,628
whether or not something is good. It

249
06:22:14,638 --> 06:22:16,310
might be reviewing code, grading an

250
06:22:16,320 --> 06:22:18,628
eval. Pick one of those and rewrite it

251
06:22:18,638 --> 06:22:20,788
as a few simple yes or no questions that

252
06:22:20,798 --> 06:22:23,270
can be answered from the evidence.

253
06:22:23,280 --> 06:22:25,430
Second is build it. This is the easy

254
06:22:25,440 --> 06:22:27,910
part. Honestly, we used Jev and ours was

255
06:22:27,920 --> 06:22:30,708
seven questions. Then the third, ship it

256
06:22:30,718 --> 06:22:33,030
in shadow mode. run it next to what you

257
06:22:33,040 --> 06:22:34,548
already have, but don't let it make the

258
06:22:34,558 --> 06:22:37,190
decisions yet. Just save it. Uh, and

259
06:22:37,200 --> 06:22:39,910
then you can analyze it later.

260
06:22:39,920 --> 06:22:42,070
Then read the cases and see where the

261
06:22:42,080 --> 06:22:44,230
two disagree. And don't overthink it.

262
06:22:44,240 --> 06:22:47,030
Just start small.

263
06:22:47,040 --> 06:22:48,788
We had trusted our judge until we

264
06:22:48,798 --> 06:22:51,910
measured it. Measure yours. Run a second

265
06:22:51,920 --> 06:22:53,830
check. Run it in shadow mode and see

266
06:22:53,840 --> 06:22:55,910
where it disagrees. You may be surprised

267
06:22:55,920 --> 06:22:59,600
with what you find. Thank you.