Machine-generated source transcript; recognition errors remain. Timestamps and subtitles use the selected source recording’s clock, not clip-relative time. Section links use cited source moments; sections without a verified moment return to the talk start.
6:24:42All
6:24:47right. Hey everyone.
6:24:50I promise I'm the last uh person on the
6:24:52hype train for Jev. Uh I'm Scott. I'm
6:24:56from uh I'm from Workday and I'm going
6:24:58to go through how we're experimenting
6:25:00with Jev uh to drive uh new innovations
6:25:04in HR and finance tasks. Uh exciting
6:25:08stuff. So, uh, at Workday, we have over
6:25:1211,000 different organizations that use
6:25:14our tools. And a lot of the problems
6:25:16that we deal with break down into kind
6:25:18of two different categories. So, one is
6:25:21very quick actions they have to take. I
6:25:23want to see my pay. I want to ask for a
6:25:25day off. The other ones are I have to
6:25:28dive into a ton of data, weed through
6:25:30it, try to figure out what's going on.
6:25:33And like many of your companies as well,
6:25:34we're trying to, you know, put agentic
6:25:36workflows around this. And so if you
6:25:39look at just a a very simple uh agentic
6:25:42loop, you know, we have a user request.
6:25:44We try to decide what to do. We're
6:25:46running tools. We're judging did we do a
6:25:48good job? Uh and then over and over
6:25:50again until we finally get a result. Uh
6:25:53now one of the problems here is that on
6:25:55each one of those we're doing LLM calls
6:25:58and there's two major problems with
6:26:00that. So one is there's too much
6:26:02intelligence. So if we look at the
6:26:04system one versus system two thinking uh
6:26:08we're doing a lot of slow deliberate
6:26:10very expensive thought processes at each
6:26:13one of those steps.
6:26:16Another problem is it's just the wrong
6:26:18tool for the job. So you know LLMs a lot
6:26:22of times are just a hammer uh looking
6:26:24for a nail. And you know we want to
6:26:26figure out like what do we actually
6:26:28need? Uh because clearly LLMs are not
6:26:31doing it. So we want something that is
6:26:34fast
6:26:35uh smart easy to use uh it's generalized
6:26:40and it's very very cheap.
6:26:44So when we look at like from a code
6:26:46perspective you know what are we really
6:26:48looking at from a from a jevlike model
6:26:50it's really just a conditional if this
6:26:53happens this should happen. Which one of
6:26:56these cases are the ones that I want to
6:26:58explore?
6:27:00And you know you've seen it among many
6:27:03of these presentations already. Uh but
6:27:05Jev has the primitives to allow us to do
6:27:07this. Uh there's the probability is it
6:27:09you know this or that and what's the
6:27:11probability? Uh what are the different
6:27:13categories that this uh particular thing
6:27:15could fall into? And then you have the
6:27:18uh the scoring mechanism which is
6:27:20basically probability and sorted.
6:27:24So you know this has been brought up as
6:27:27well. You know, my first thought when I
6:27:29saw, you know, the Jev models was, well,
6:27:31we already have classifiers. I've been
6:27:33doing this for a long time. How is it
6:27:35actually different?
6:27:37And as many of the speakers have talked
6:27:39about already, you know, with
6:27:40classifiers, I need to figure out, you
6:27:43know, what's my golden data set. I have
6:27:46to train it. New data comes out. I have
6:27:48to iterate over and over. With Jev, just
6:27:51out of the box, it just worked. We
6:27:53plugged in our question. We classified
6:27:56it. we didn't really have to do
6:27:57anything. It generalized to our
6:27:58question.
6:28:00Uh and so we started using it for a lot
6:28:02of the use cases that you've seen from
6:28:04some of the presenters already. Uh so we
6:28:06use it for routing. That's kind of the
6:28:07canonical example. Um you know figuring
6:28:10out did this step work you know as a
6:28:12judge. Uh we looked at uh you know
6:28:16resumes you know is this person
6:28:19qualified for ML work? Uh, and I want to
6:28:23do kind of an overly simplified example
6:28:25as well. Um, so no one taught Jev like
6:28:28what a fish is. And so this is actually
6:28:30a question we asked like is a fish a
6:28:32pet? Well, 50/50 might be a pet
6:28:35sometimes, might not be. Um, and so, you
6:28:38know, if I ran into this, then I have to
6:28:40figure out um, what do I do next? Maybe
6:28:43I can give it some more context. Oh, is
6:28:45a fishing a bowl a pet? You know, and
6:28:47after 20 milliseconds, I can ask this
6:28:49again. All right, that's great. Oh, is a
6:28:52fish on my plate a pet? Clearly not. Uh,
6:28:55so this is something that wasn't trained
6:28:57out of the box in seconds. It just
6:28:59worked.
6:29:02All right, so now on to some of the
6:29:03demos that we're do that um, you know,
6:29:06we've done at workday. So just a caveat
6:29:09on some of these. Um, it's not real
6:29:12data, so we're not using any of your you
6:29:14know, resumes or payroll data. Uh, these
6:29:16are also not in production now and may
6:29:18or may not be in the future. So that's
6:29:20something our legal asked me to say. So
6:29:22let's get into, you know, some of the uh
6:29:25the demos.
6:29:27All right. So starting with
6:29:28autocomplete. So autocomplete starts to
6:29:31predict what you're trying to ask. It's
6:29:34just predicting the question. It's not
6:29:35helping you get to the answer. So as you
6:29:38start to type more, it's going to guess
6:29:40like, oh, I want to figure out if this
6:29:43person wants to take a day off work.
6:29:44Great. You've guessed the question, but
6:29:46you really haven't, you know, helped me
6:29:48get to the answer.
6:29:50So what we want to do is move from just
6:29:52predicting the word or the next block of
6:29:54code and actually help you finish the
6:29:56task. And so this is best shown with a a
6:30:00demo that we have where we go from
6:30:02predicting what you want to say to
6:30:05predicting what you want to do. So
6:30:08hopefully the video play. There we go.
6:30:10All right. So I need to take time off of
6:30:13off of uh next week. And as I start to
6:30:17ask this, Jev is actually figuring out
6:30:19under the hood, okay, what what more
6:30:22context do I have? How can I adjust this
6:30:25UI based on the information that I'm
6:30:28that I'm getting? This is not something
6:30:30that we could have done with an LLM.
6:30:31We're getting, you know, we're we're
6:30:33doing this, you know, in a loop while
6:30:35someone's asking. Um the pay slip, you
6:30:39know, you shouldn't have to go in and
6:30:40ask an agent, hey, can you go find out
6:30:42how much I was paid? Uh what you can do
6:30:45is just in the search say what was my
6:30:47pay last period
6:30:50pops up an interface allows you to
6:30:53unhide it you know see what your pay was
6:30:55in the last period very simple gets you
6:30:59to the action book a meeting.
6:31:02So I know that I want to book a meeting
6:31:05and uh I'm going to want to do it for
6:31:08next Friday. Let me do it at 2 PM. Oh, I
6:31:12want to do it with my boss. Make sure
6:31:14that I get my, you know, my boss on
6:31:16this. Uh, Logan McNeel. There we go.
6:31:19Meeting's booked.
6:31:21And then a final one that we have just
6:31:23for this example, but there's many more.
6:31:26Uh, is finding a person. So, in this
6:31:29one, I may have some semantic knowledge
6:31:31like find the person who owns this part
6:31:34of the organization or just like who's
6:31:37Ashanti? I want to contact her. uh pop
6:31:39it up. Click on email. Good to go.
6:31:46All right. So, what's happening under
6:31:47the hood? So, because Jev is fast and
6:31:50cheap, uh just like a type ahead, we can
6:31:54start to render a UI on this on that
6:31:56same cadence, we can look at as the
6:31:59words are coming, do a prediction of
6:32:01intent and start to render an interface.
6:32:05And you know we could do this at you
6:32:07know about 20 bucks for a million
6:32:08judgments. Uh and the way that we're
6:32:11doing it is kind of twofold um or two
6:32:14parts I should say. So first Jev is
6:32:17picking the card of saying what's the
6:32:18intent. All right there's a card that's
6:32:20going to help you out and then the team
6:32:22that's responsible in that domain then
6:32:24can decide what are the questions to
6:32:26progressively fill out that card. And so
6:32:29there's sort of this two-step process.
6:32:31Uh the first step is getting people in
6:32:33the front door, getting the right card
6:32:35up, and then the second one is making
6:32:37sure that you fill in that card to help
6:32:39them accomplish the task that they're
6:32:41after. Now, we have an eval on here
6:32:44before we merge any card. We want to
6:32:46make sure that it doesn't collide with
6:32:47any other one. And at least in our
6:32:50initial experiments, you know, we we've
6:32:52done seven cards or a just chat because
6:32:55it should if there's no card to actually
6:32:57solve it, it should allow you to just
6:32:58continue chatting. uh we were able to
6:33:00get you know 96% accuracy on that.
6:33:05All right. So now on to the recruiter.
6:33:07So let's talk about the recruiter
6:33:08problem. All right. So this is not like
6:33:10a fast uh task. It's something where
6:33:13recruiter has a pile of hundreds of
6:33:15resumes and has to figure out who has
6:33:18the best resume to accomplish the uh or
6:33:21fill the wreck that they're trying to
6:33:22fill. Now I first want to look at like
6:33:25how we would have done this before. uh
6:33:27we would have done you know an embedding
6:33:29put the vectors in a vector DB then had
6:33:33a retrieval layer then have a reranker
6:33:35and then figure out based on the
6:33:37reranker which ones that we want to put
6:33:39in front of uh the recruiter. Uh one
6:33:42interesting call out is the Quinn
6:33:44reranker uh looks surprisingly similar
6:33:47to uh the uh the Jev process as well. Uh
6:33:52and so that that's kind of an
6:33:53interesting correlary if you've used
6:33:54that one before. Uh but with Jev, we
6:33:58basically take out all the embedding
6:33:59steps. I don't have to train anything
6:34:02up. I don't have to do embedding. I
6:34:03don't have to put things in a vector
6:34:05database. Um what I can do is just do a
6:34:10function on my database. So put all the
6:34:13resumes into a database. Uh use natural
6:34:17language to query it, iterate over every
6:34:20single one of them, and then just figure
6:34:22out which ones match my criteria.
6:34:25So uh I'll go into an uh you know an
6:34:28actual demo of this exact same one. So
6:34:31here we have a select statement and so
6:34:33this is against a duct db where we have
6:34:36a function that is going to call jev to
6:34:40figure out what's the probability that
6:34:41someone matches this criteria. We're
6:34:43going to get it a score and then we're
6:34:45going to order by that score. Now I'm
6:34:46going to do about 20 and as I said these
6:34:49are all fakes and none of your resumes
6:34:50are in here I promise. And then here we
6:34:53see uh these are very uh ML focused uh
6:34:56synthetic data as well. So it's not a
6:34:59surprise that we would have uh very high
6:35:01hit rates against someone who's worked
6:35:03in fraud, trust, safety problems. Um but
6:35:06let's look at let's change this
6:35:07question. Let's uh be a little bit more
6:35:09adversarial on this one. So a lot of
6:35:12these résumés have the word pipeline in
6:35:14them. So whether it's a data pipeline or
6:35:17an AI pipeline, uh pipe's going to come
6:35:20out. So let's just ask like is this
6:35:22person qualified to do plumbing? If you
6:35:23were going to just do a lexical search,
6:35:25it might pick up some of these. So we
6:35:28put it into Jev and asked, you know, of
6:35:29all these ML res resume résumés, let's,
6:35:32you know, trim the limit down to
6:35:34five here. Uh go ahead and run it.
6:35:38And we can see, you know, that um very
6:35:43very low probability on these. So these
6:35:45are ones that we would almost definitely
6:35:47throw out. And you can see the speed. So
6:35:49we went over 117 records. Uh we had 2.3
6:35:53seconds of wall time and spend almost
6:35:55nothing on this query to just iterate
6:35:57over everything. Uh so Jev really kind
6:36:00of changed the game for us because every
6:36:02every place where we're thinking either
6:36:04either this is too expensive or it's too
6:36:06slow, we thought of new places to inject
6:36:09Jev into our pipeline.
6:36:11And so [snorts] just to summarize, you
6:36:12know, the two areas that I went through
6:36:15are one, where can we decide very
6:36:18quickly and get to the intent and get
6:36:20the person on their way? And then the
6:36:23second one is to cut down on some of
6:36:25the, you know, really deep work. How do
6:36:28we filter those out for, you know, our
6:36:30customers? And so I hope this that that
6:36:33this gave you a little bit of insight
6:36:34into the things that you can take back
6:36:36to you know your own companies and how
6:36:38and you know imagine the things that you
6:36:40can do with Jev and decision models as
6:36:42well. And that's my time. Thank you very
6:36:44much.
Original stream, 6:35:20. Roberts changes the question in the query, including a deliberately different plumbing criterion.