1
06:24:42,400 --> 06:24:47,190
All

2
06:24:47,200 --> 06:24:50,070
right. Hey everyone.

3
06:24:50,080 --> 06:24:52,470
I promise I'm the last uh person on the

4
06:24:52,480 --> 06:24:56,390
hype train for Jev. Uh I'm Scott. I'm

5
06:24:56,400 --> 06:24:58,868
from uh I'm from Workday and I'm going

6
06:24:58,878 --> 06:25:00,708
to go through how we're experimenting

7
06:25:00,718 --> 06:25:04,390
with Jev uh to drive uh new innovations

8
06:25:04,400 --> 06:25:08,310
in HR and finance tasks. Uh exciting

9
06:25:08,320 --> 06:25:11,990
stuff. So, uh, at Workday, we have over

10
06:25:12,000 --> 06:25:14,150
11,000 different organizations that use

11
06:25:14,160 --> 06:25:16,548
our tools. And a lot of the problems

12
06:25:16,558 --> 06:25:18,948
that we deal with break down into kind

13
06:25:18,958 --> 06:25:21,110
of two different categories. So, one is

14
06:25:21,120 --> 06:25:23,510
very quick actions they have to take. I

15
06:25:23,520 --> 06:25:25,270
want to see my pay. I want to ask for a

16
06:25:25,280 --> 06:25:27,990
day off. The other ones are I have to

17
06:25:28,000 --> 06:25:30,628
dive into a ton of data, weed through

18
06:25:30,638 --> 06:25:33,110
it, try to figure out what's going on.

19
06:25:33,120 --> 06:25:34,788
And like many of your companies as well,

20
06:25:34,798 --> 06:25:36,708
we're trying to, you know, put agentic

21
06:25:36,718 --> 06:25:39,190
workflows around this. And so if you

22
06:25:39,200 --> 06:25:42,230
look at just a a very simple uh agentic

23
06:25:42,240 --> 06:25:44,708
loop, you know, we have a user request.

24
06:25:44,718 --> 06:25:46,470
We try to decide what to do. We're

25
06:25:46,480 --> 06:25:48,390
running tools. We're judging did we do a

26
06:25:48,400 --> 06:25:50,788
good job? Uh and then over and over

27
06:25:50,798 --> 06:25:53,910
again until we finally get a result. Uh

28
06:25:53,920 --> 06:25:55,750
now one of the problems here is that on

29
06:25:55,760 --> 06:25:58,708
each one of those we're doing LLM calls

30
06:25:58,718 --> 06:26:00,390
and there's two major problems with

31
06:26:00,400 --> 06:26:02,708
that. So one is there's too much

32
06:26:02,718 --> 06:26:04,948
intelligence. So if we look at the

33
06:26:04,958 --> 06:26:08,310
system one versus system two thinking uh

34
06:26:08,320 --> 06:26:10,470
we're doing a lot of slow deliberate

35
06:26:10,480 --> 06:26:13,270
very expensive thought processes at each

36
06:26:13,280 --> 06:26:16,390
one of those steps.

37
06:26:16,400 --> 06:26:18,470
Another problem is it's just the wrong

38
06:26:18,480 --> 06:26:21,990
tool for the job. So you know LLMs a lot

39
06:26:22,000 --> 06:26:23,990
of times are just a hammer uh looking

40
06:26:24,000 --> 06:26:26,868
for a nail. And you know we want to

41
06:26:26,878 --> 06:26:28,708
figure out like what do we actually

42
06:26:28,718 --> 06:26:31,190
need? Uh because clearly LLMs are not

43
06:26:31,200 --> 06:26:34,070
doing it. So we want something that is

44
06:26:34,080 --> 06:26:35,590
fast

45
06:26:35,600 --> 06:26:40,868
uh smart easy to use uh it's generalized

46
06:26:40,878 --> 06:26:44,628
and it's very very cheap.

47
06:26:44,638 --> 06:26:46,548
So when we look at like from a code

48
06:26:46,558 --> 06:26:48,230
perspective you know what are we really

49
06:26:48,240 --> 06:26:50,708
looking at from a from a jevlike model

50
06:26:50,718 --> 06:26:53,190
it's really just a conditional if this

51
06:26:53,200 --> 06:26:56,150
happens this should happen. Which one of

52
06:26:56,160 --> 06:26:58,788
these cases are the ones that I want to

53
06:26:58,798 --> 06:27:00,548
explore?

54
06:27:00,558 --> 06:27:03,030
And you know you've seen it among many

55
06:27:03,040 --> 06:27:05,110
of these presentations already. Uh but

56
06:27:05,120 --> 06:27:07,190
Jev has the primitives to allow us to do

57
06:27:07,200 --> 06:27:09,910
this. Uh there's the probability is it

58
06:27:09,920 --> 06:27:11,590
you know this or that and what's the

59
06:27:11,600 --> 06:27:13,750
probability? Uh what are the different

60
06:27:13,760 --> 06:27:15,910
categories that this uh particular thing

61
06:27:15,920 --> 06:27:18,628
could fall into? And then you have the

62
06:27:18,638 --> 06:27:20,150
uh the scoring mechanism which is

63
06:27:20,160 --> 06:27:24,628
basically probability and sorted.

64
06:27:24,638 --> 06:27:27,350
So you know this has been brought up as

65
06:27:27,360 --> 06:27:29,030
well. You know, my first thought when I

66
06:27:29,040 --> 06:27:31,430
saw, you know, the Jev models was, well,

67
06:27:31,440 --> 06:27:33,350
we already have classifiers. I've been

68
06:27:33,360 --> 06:27:35,430
doing this for a long time. How is it

69
06:27:35,440 --> 06:27:37,430
actually different?

70
06:27:37,440 --> 06:27:39,030
And as many of the speakers have talked

71
06:27:39,040 --> 06:27:40,310
about already, you know, with

72
06:27:40,320 --> 06:27:43,510
classifiers, I need to figure out, you

73
06:27:43,520 --> 06:27:45,990
know, what's my golden data set. I have

74
06:27:46,000 --> 06:27:48,868
to train it. New data comes out. I have

75
06:27:48,878 --> 06:27:51,750
to iterate over and over. With Jev, just

76
06:27:51,760 --> 06:27:53,910
out of the box, it just worked. We

77
06:27:53,920 --> 06:27:55,990
plugged in our question. We classified

78
06:27:56,000 --> 06:27:57,350
it. we didn't really have to do

79
06:27:57,360 --> 06:27:58,948
anything. It generalized to our

80
06:27:58,958 --> 06:28:00,468
question.

81
06:28:00,478 --> 06:28:02,390
Uh and so we started using it for a lot

82
06:28:02,400 --> 06:28:04,230
of the use cases that you've seen from

83
06:28:04,240 --> 06:28:06,230
some of the presenters already. Uh so we

84
06:28:06,240 --> 06:28:07,590
use it for routing. That's kind of the

85
06:28:07,600 --> 06:28:10,548
canonical example. Um you know figuring

86
06:28:10,558 --> 06:28:12,788
out did this step work you know as a

87
06:28:12,798 --> 06:28:16,948
judge. Uh we looked at uh you know

88
06:28:16,958 --> 06:28:19,190
resumes you know is this person

89
06:28:19,200 --> 06:28:23,670
qualified for ML work? Uh, and I want to

90
06:28:23,680 --> 06:28:25,590
do kind of an overly simplified example

91
06:28:25,600 --> 06:28:28,708
as well. Um, so no one taught Jev like

92
06:28:28,718 --> 06:28:30,708
what a fish is. And so this is actually

93
06:28:30,718 --> 06:28:32,548
a question we asked like is a fish a

94
06:28:32,558 --> 06:28:35,270
pet? Well, 50/50 might be a pet

95
06:28:35,280 --> 06:28:38,948
sometimes, might not be. Um, and so, you

96
06:28:38,958 --> 06:28:40,868
know, if I ran into this, then I have to

97
06:28:40,878 --> 06:28:43,750
figure out um, what do I do next? Maybe

98
06:28:43,760 --> 06:28:45,510
I can give it some more context. Oh, is

99
06:28:45,520 --> 06:28:47,670
a fishing a bowl a pet? You know, and

100
06:28:47,680 --> 06:28:49,350
after 20 milliseconds, I can ask this

101
06:28:49,360 --> 06:28:51,990
again. All right, that's great. Oh, is a

102
06:28:52,000 --> 06:28:55,670
fish on my plate a pet? Clearly not. Uh,

103
06:28:55,680 --> 06:28:57,430
so this is something that wasn't trained

104
06:28:57,440 --> 06:28:59,910
out of the box in seconds. It just

105
06:28:59,920 --> 06:29:01,990
worked.

106
06:29:02,000 --> 06:29:03,430
All right, so now on to some of the

107
06:29:03,440 --> 06:29:06,150
demos that we're do that um, you know,

108
06:29:06,160 --> 06:29:09,670
we've done at workday. So just a caveat

109
06:29:09,680 --> 06:29:12,230
on some of these. Um, it's not real

110
06:29:12,240 --> 06:29:14,070
data, so we're not using any of your you

111
06:29:14,080 --> 06:29:16,628
know, resumes or payroll data. Uh, these

112
06:29:16,638 --> 06:29:18,468
are also not in production now and may

113
06:29:18,478 --> 06:29:20,548
or may not be in the future. So that's

114
06:29:20,558 --> 06:29:22,868
something our legal asked me to say. So

115
06:29:22,878 --> 06:29:25,110
let's get into, you know, some of the uh

116
06:29:25,120 --> 06:29:27,030
the demos.

117
06:29:27,040 --> 06:29:28,628
All right. So starting with

118
06:29:28,638 --> 06:29:31,430
autocomplete. So autocomplete starts to

119
06:29:31,440 --> 06:29:33,990
predict what you're trying to ask. It's

120
06:29:34,000 --> 06:29:35,830
just predicting the question. It's not

121
06:29:35,840 --> 06:29:38,230
helping you get to the answer. So as you

122
06:29:38,240 --> 06:29:40,150
start to type more, it's going to guess

123
06:29:40,160 --> 06:29:43,270
like, oh, I want to figure out if this

124
06:29:43,280 --> 06:29:44,948
person wants to take a day off work.

125
06:29:44,958 --> 06:29:46,230
Great. You've guessed the question, but

126
06:29:46,240 --> 06:29:48,230
you really haven't, you know, helped me

127
06:29:48,240 --> 06:29:49,990
get to the answer.

128
06:29:50,000 --> 06:29:52,230
So what we want to do is move from just

129
06:29:52,240 --> 06:29:54,708
predicting the word or the next block of

130
06:29:54,718 --> 06:29:56,868
code and actually help you finish the

131
06:29:56,878 --> 06:30:00,230
task. And so this is best shown with a a

132
06:30:00,240 --> 06:30:02,788
demo that we have where we go from

133
06:30:02,798 --> 06:30:05,510
predicting what you want to say to

134
06:30:05,520 --> 06:30:08,150
predicting what you want to do. So

135
06:30:08,160 --> 06:30:10,070
hopefully the video play. There we go.

136
06:30:10,080 --> 06:30:13,510
All right. So I need to take time off of

137
06:30:13,520 --> 06:30:17,270
off of uh next week. And as I start to

138
06:30:17,280 --> 06:30:19,590
ask this, Jev is actually figuring out

139
06:30:19,600 --> 06:30:22,150
under the hood, okay, what what more

140
06:30:22,160 --> 06:30:25,030
context do I have? How can I adjust this

141
06:30:25,040 --> 06:30:28,150
UI based on the information that I'm

142
06:30:28,160 --> 06:30:29,990
that I'm getting? This is not something

143
06:30:30,000 --> 06:30:31,670
that we could have done with an LLM.

144
06:30:31,680 --> 06:30:33,350
We're getting, you know, we're we're

145
06:30:33,360 --> 06:30:35,510
doing this, you know, in a loop while

146
06:30:35,520 --> 06:30:39,030
someone's asking. Um the pay slip, you

147
06:30:39,040 --> 06:30:40,708
know, you shouldn't have to go in and

148
06:30:40,718 --> 06:30:42,548
ask an agent, hey, can you go find out

149
06:30:42,558 --> 06:30:45,350
how much I was paid? Uh what you can do

150
06:30:45,360 --> 06:30:47,830
is just in the search say what was my

151
06:30:47,840 --> 06:30:50,070
pay last period

152
06:30:50,080 --> 06:30:53,190
pops up an interface allows you to

153
06:30:53,200 --> 06:30:55,910
unhide it you know see what your pay was

154
06:30:55,920 --> 06:30:59,110
in the last period very simple gets you

155
06:30:59,120 --> 06:31:02,948
to the action book a meeting.

156
06:31:02,958 --> 06:31:05,430
So I know that I want to book a meeting

157
06:31:05,440 --> 06:31:08,390
and uh I'm going to want to do it for

158
06:31:08,400 --> 06:31:12,948
next Friday. Let me do it at 2 PM. Oh, I

159
06:31:12,958 --> 06:31:14,548
want to do it with my boss. Make sure

160
06:31:14,558 --> 06:31:16,868
that I get my, you know, my boss on

161
06:31:16,878 --> 06:31:19,910
this. Uh, Logan McNeel. There we go.

162
06:31:19,920 --> 06:31:21,910
Meeting's booked.

163
06:31:21,920 --> 06:31:23,510
And then a final one that we have just

164
06:31:23,520 --> 06:31:25,990
for this example, but there's many more.

165
06:31:26,000 --> 06:31:29,670
Uh, is finding a person. So, in this

166
06:31:29,680 --> 06:31:31,750
one, I may have some semantic knowledge

167
06:31:31,760 --> 06:31:34,150
like find the person who owns this part

168
06:31:34,160 --> 06:31:37,030
of the organization or just like who's

169
06:31:37,040 --> 06:31:39,830
Ashanti? I want to contact her. uh pop

170
06:31:39,840 --> 06:31:46,070
it up. Click on email. Good to go.

171
06:31:46,080 --> 06:31:47,190
All right. So, what's happening under

172
06:31:47,200 --> 06:31:50,708
the hood? So, because Jev is fast and

173
06:31:50,718 --> 06:31:54,310
cheap, uh just like a type ahead, we can

174
06:31:54,320 --> 06:31:56,948
start to render a UI on this on that

175
06:31:56,958 --> 06:31:59,510
same cadence, we can look at as the

176
06:31:59,520 --> 06:32:01,750
words are coming, do a prediction of

177
06:32:01,760 --> 06:32:05,110
intent and start to render an interface.

178
06:32:05,120 --> 06:32:07,030
And you know we could do this at you

179
06:32:07,040 --> 06:32:08,548
know about 20 bucks for a million

180
06:32:08,558 --> 06:32:11,030
judgments. Uh and the way that we're

181
06:32:11,040 --> 06:32:14,230
doing it is kind of twofold um or two

182
06:32:14,240 --> 06:32:17,030
parts I should say. So first Jev is

183
06:32:17,040 --> 06:32:18,948
picking the card of saying what's the

184
06:32:18,958 --> 06:32:20,548
intent. All right there's a card that's

185
06:32:20,558 --> 06:32:22,788
going to help you out and then the team

186
06:32:22,798 --> 06:32:24,628
that's responsible in that domain then

187
06:32:24,638 --> 06:32:26,708
can decide what are the questions to

188
06:32:26,718 --> 06:32:29,030
progressively fill out that card. And so

189
06:32:29,040 --> 06:32:31,750
there's sort of this two-step process.

190
06:32:31,760 --> 06:32:33,670
Uh the first step is getting people in

191
06:32:33,680 --> 06:32:35,350
the front door, getting the right card

192
06:32:35,360 --> 06:32:37,510
up, and then the second one is making

193
06:32:37,520 --> 06:32:39,830
sure that you fill in that card to help

194
06:32:39,840 --> 06:32:41,670
them accomplish the task that they're

195
06:32:41,680 --> 06:32:44,628
after. Now, we have an eval on here

196
06:32:44,638 --> 06:32:46,390
before we merge any card. We want to

197
06:32:46,400 --> 06:32:47,830
make sure that it doesn't collide with

198
06:32:47,840 --> 06:32:50,310
any other one. And at least in our

199
06:32:50,320 --> 06:32:52,070
initial experiments, you know, we we've

200
06:32:52,080 --> 06:32:55,190
done seven cards or a just chat because

201
06:32:55,200 --> 06:32:57,110
it should if there's no card to actually

202
06:32:57,120 --> 06:32:58,468
solve it, it should allow you to just

203
06:32:58,478 --> 06:33:00,708
continue chatting. uh we were able to

204
06:33:00,718 --> 06:33:05,270
get you know 96% accuracy on that.

205
06:33:05,280 --> 06:33:07,030
All right. So now on to the recruiter.

206
06:33:07,040 --> 06:33:08,390
So let's talk about the recruiter

207
06:33:08,400 --> 06:33:10,070
problem. All right. So this is not like

208
06:33:10,080 --> 06:33:13,510
a fast uh task. It's something where

209
06:33:13,520 --> 06:33:15,750
recruiter has a pile of hundreds of

210
06:33:15,760 --> 06:33:18,070
resumes and has to figure out who has

211
06:33:18,080 --> 06:33:21,030
the best resume to accomplish the uh or

212
06:33:21,040 --> 06:33:22,310
fill the wreck that they're trying to

213
06:33:22,320 --> 06:33:25,190
fill. Now I first want to look at like

214
06:33:25,200 --> 06:33:27,350
how we would have done this before. uh

215
06:33:27,360 --> 06:33:29,910
we would have done you know an embedding

216
06:33:29,920 --> 06:33:33,110
put the vectors in a vector DB then had

217
06:33:33,120 --> 06:33:35,910
a retrieval layer then have a reranker

218
06:33:35,920 --> 06:33:37,430
and then figure out based on the

219
06:33:37,440 --> 06:33:39,190
reranker which ones that we want to put

220
06:33:39,200 --> 06:33:42,468
in front of uh the recruiter. Uh one

221
06:33:42,478 --> 06:33:44,788
interesting call out is the Quinn

222
06:33:44,798 --> 06:33:47,510
reranker uh looks surprisingly similar

223
06:33:47,520 --> 06:33:51,990
to uh the uh the Jev process as well. Uh

224
06:33:52,000 --> 06:33:53,350
and so that that's kind of an

225
06:33:53,360 --> 06:33:54,788
interesting correlary if you've used

226
06:33:54,798 --> 06:33:58,150
that one before. Uh but with Jev, we

227
06:33:58,160 --> 06:33:59,910
basically take out all the embedding

228
06:33:59,920 --> 06:34:02,070
steps. I don't have to train anything

229
06:34:02,080 --> 06:34:03,830
up. I don't have to do embedding. I

230
06:34:03,840 --> 06:34:05,190
don't have to put things in a vector

231
06:34:05,200 --> 06:34:10,468
database. Um what I can do is just do a

232
06:34:10,478 --> 06:34:13,430
function on my database. So put all the

233
06:34:13,440 --> 06:34:17,510
resumes into a database. Uh use natural

234
06:34:17,520 --> 06:34:20,230
language to query it, iterate over every

235
06:34:20,240 --> 06:34:22,948
single one of them, and then just figure

236
06:34:22,958 --> 06:34:25,910
out which ones match my criteria.

237
06:34:25,920 --> 06:34:28,310
So uh I'll go into an uh you know an

238
06:34:28,320 --> 06:34:31,350
actual demo of this exact same one. So

239
06:34:31,360 --> 06:34:33,030
here we have a select statement and so

240
06:34:33,040 --> 06:34:35,990
this is against a duct db where we have

241
06:34:36,000 --> 06:34:39,990
a function that is going to call jev to

242
06:34:40,000 --> 06:34:41,270
figure out what's the probability that

243
06:34:41,280 --> 06:34:43,670
someone matches this criteria. We're

244
06:34:43,680 --> 06:34:45,030
going to get it a score and then we're

245
06:34:45,040 --> 06:34:46,708
going to order by that score. Now I'm

246
06:34:46,718 --> 06:34:49,030
going to do about 20 and as I said these

247
06:34:49,040 --> 06:34:50,948
are all fakes and none of your resumes

248
06:34:50,958 --> 06:34:53,590
are in here I promise. And then here we

249
06:34:53,600 --> 06:34:56,788
see uh these are very uh ML focused uh

250
06:34:56,798 --> 06:34:59,350
synthetic data as well. So it's not a

251
06:34:59,360 --> 06:35:01,510
surprise that we would have uh very high

252
06:35:01,520 --> 06:35:03,190
hit rates against someone who's worked

253
06:35:03,200 --> 06:35:06,468
in fraud, trust, safety problems. Um but

254
06:35:06,478 --> 06:35:07,910
let's look at let's change this

255
06:35:07,920 --> 06:35:09,830
question. Let's uh be a little bit more

256
06:35:09,840 --> 06:35:12,628
adversarial on this one. So a lot of

257
06:35:12,638 --> 06:35:14,390
these résumés have the word pipeline in

258
06:35:14,400 --> 06:35:17,590
them. So whether it's a data pipeline or

259
06:35:17,600 --> 06:35:20,390
an AI pipeline, uh pipe's going to come

260
06:35:20,400 --> 06:35:22,310
out. So let's just ask like is this

261
06:35:22,320 --> 06:35:23,910
person qualified to do plumbing? If you

262
06:35:23,920 --> 06:35:25,830
were going to just do a lexical search,

263
06:35:25,840 --> 06:35:27,990
it might pick up some of these. So we

264
06:35:28,000 --> 06:35:29,830
put it into Jev and asked, you know, of

265
06:35:29,840 --> 06:35:32,150
all these ML res resume résumés, let's,

266
06:35:32,160 --> 06:35:34,948
you know, trim the limit down to

267
06:35:34,958 --> 06:35:38,948
five here. Uh go ahead and run it.

268
06:35:38,958 --> 06:35:43,510
And we can see, you know, that um very

269
06:35:43,520 --> 06:35:45,750
very low probability on these. So these

270
06:35:45,760 --> 06:35:47,190
are ones that we would almost definitely

271
06:35:47,200 --> 06:35:49,430
throw out. And you can see the speed. So

272
06:35:49,440 --> 06:35:53,030
we went over 117 records. Uh we had 2.3

273
06:35:53,040 --> 06:35:55,830
seconds of wall time and spend almost

274
06:35:55,840 --> 06:35:57,430
nothing on this query to just iterate

275
06:35:57,440 --> 06:36:00,868
over everything. Uh so Jev really kind

276
06:36:00,878 --> 06:36:02,868
of changed the game for us because every

277
06:36:02,878 --> 06:36:04,548
every place where we're thinking either

278
06:36:04,558 --> 06:36:06,548
either this is too expensive or it's too

279
06:36:06,558 --> 06:36:09,030
slow, we thought of new places to inject

280
06:36:09,040 --> 06:36:11,270
Jev into our pipeline.

281
06:36:11,280 --> 06:36:12,948
And so [snorts] just to summarize, you

282
06:36:12,958 --> 06:36:15,430
know, the two areas that I went through

283
06:36:15,440 --> 06:36:18,070
are one, where can we decide very

284
06:36:18,080 --> 06:36:20,948
quickly and get to the intent and get

285
06:36:20,958 --> 06:36:23,110
the person on their way? And then the

286
06:36:23,120 --> 06:36:25,510
second one is to cut down on some of

287
06:36:25,520 --> 06:36:28,070
the, you know, really deep work. How do

288
06:36:28,080 --> 06:36:30,230
we filter those out for, you know, our

289
06:36:30,240 --> 06:36:33,270
customers? And so I hope this that that

290
06:36:33,280 --> 06:36:34,628
this gave you a little bit of insight

291
06:36:34,638 --> 06:36:36,070
into the things that you can take back

292
06:36:36,080 --> 06:36:38,150
to you know your own companies and how

293
06:36:38,160 --> 06:36:40,070
and you know imagine the things that you

294
06:36:40,080 --> 06:36:41,990
can do with Jev and decision models as

295
06:36:42,000 --> 06:36:44,628
well. And that's my time. Thank you very

296
06:36:44,638 --> 06:36:46,200
much.