Your Automation Is Overthinking It

5:52:19I'm joined on stage by Kevin. Uh we're

5:52:22here to talk about how we believe that

5:52:25uh your automations are overthinking it.

5:52:28So um tell us a little bit about

5:52:30ourselves. We are from a company called

5:52:32UiPath. Uh we have been doing automation

5:52:35for the last 20 years. uh we've been

5:52:37doing uh RPA before and uh we service

5:52:40more than 10,000 organizations around

5:52:42the around the world. Um and we build uh

5:52:45an agentic process orchestration layer

5:52:48which basically brings agents, robots

5:52:51and people together to interact with

5:52:53each other and uh build extremely

5:52:55scalable uh automations that run for

5:52:58days and weeks. So we were inspired by

5:53:02this particular slide that you're seeing

5:53:04right now and you saw today today before

5:53:06us as well. Uh when uh Diego presented

5:53:10it 3 months ago the same slide uh it

5:53:13resonated really well with us because

5:53:16this is how we see automation as well.

5:53:19So the traditional software was is

5:53:22basically RPA. It's just robotic process

5:53:25automation if else just executes some

5:53:27task and then keeps going. And the

5:53:28control flow is deterministic. It just

5:53:30keeps going. And then when the LLMs came

5:53:33out, uh we all raced to go and build the

5:53:36just in time software where we have this

5:53:38one agent. We give it a bunch of tools.

5:53:40We give it a task and ask it to finish.

5:53:43And then we were seeing good results,

5:53:45bad results, and then we added

5:53:47evaluations to slowly build the results

5:53:49up. And then finally we landed on

5:53:52agentic process orchestration which kind

5:53:54of weaves in this cognitive decision

5:53:57through a logical uh decision layer. So

5:54:01you have deterministic calls and clear

5:54:04cognitive decision points where you

5:54:06would use an LLM to control the control

5:54:09flow rather than leaving it all in the

5:54:11hands of an agent. So at this time we we

5:54:15saw where J would fit in and when uh

5:54:17Diego was talking about Je we were

5:54:19really excited because we saw Je fit in

5:54:22in a very particular niche area. So u

5:54:26here we see that Je is a very powerful

5:54:30tool as Diego was saying it's a

5:54:33foundational tool that we would be able

5:54:35to integrate into all automation. Uh and

5:54:37it fit right in between a deterministic

5:54:40case and an agent case. So, Jev is

5:54:44extremely fast and most of you guys

5:54:46would have tried it and we were

5:54:48surprised by how fast and how parallel

5:54:50we could make it work and uh and how

5:54:53intelligent it was. So, we were able to

5:54:55benchmark it around how good it was

5:54:57compared to an LLM and was pretty good

5:54:59uh not as a full LM but it was pretty

5:55:02good at in terms of intelligence as

5:55:03well. So it falls in between and there

5:55:06are so many ways we could use je and uh

5:55:09I want to hand it off to Kevin to take

5:55:11you uh take you guys through a scenario

5:55:13where uh we use Jev in uh UiPath Maestro

5:55:16Flow.

5:55:17>> Yeah.

5:55:18>> Thanks Chibi. Um so hi everybody. Uh

5:55:21today I want to actually walk us through

5:55:22a little bit about what UiPath is

5:55:24building with UiPath Maestro Flow and

5:55:26talk through a little bit of how we

5:55:29would go through and coordinate humans,

5:55:31robots, and agents all together inside

5:55:33of one canvas. Uh what we're going to

5:55:35walk through today is essentially um you

5:55:37know essentially a flow that will

5:55:39demonstrate how we kind of weave all

5:55:41these things together. And the way that

5:55:43we're approaching this is that we're

5:55:44going to end up having system one agents

5:55:46which are going to be able to make quick

5:55:48intelligence happen within our process.

5:55:50Think about things like, hey, is this

5:55:52transaction fraud or not? Um,

5:55:55additionally, we'll have u system 2

5:55:58agents running where we're going to have

5:55:59more of methodical deeper investigation

5:56:01that's going on where agents are going

5:56:03off and calling tools. And we'll see

5:56:05kind of this pattern these patterns

5:56:06start to emerge where we're able to hand

5:56:08off from one to the next. And so with

5:56:10that, I'll I'll jump into an actual demo

5:56:12where we're going to demonstrate a

5:56:14account fraud uh scenario. And so our

5:56:18demo starts here with uh effectively a

5:56:22transaction coming in for $48,000.

5:56:25And this wasn't something that we were

5:56:27expecting. And so uh as I'm reviewing my

5:56:29bank portal here, I'm going to go ahead

5:56:30and report this as a fraudulent

5:56:32transaction. And so now this would end

5:56:34up smitting it off into our UiPath

5:56:36Maestro flow where this would act as

5:56:38essentially the trigger for this

5:56:40endto-end workflow to to go off and

5:56:42start processing. What we're looking at

5:56:44is a collection of various different

5:56:46nodes that are each going to kind of uh

5:56:49be orchestrated from the left side of

5:56:51the canvas over to the right. And we're

5:56:53going to use uh essentially a

5:56:54combination of system one and system two

5:56:56models to actually process this uh and

5:56:58do analysis of this is fraud or not. And

5:57:01so the first thing that actually takes

5:57:02place is we're going to go through and

5:57:04get account activity. We're going to

5:57:07send this off to Jev to basically pull

5:57:1060 transactions from the most recent uh

5:57:12credit card purchases that were made and

5:57:14analyze each one of those individually

5:57:16whether or not these are fraud. And this

5:57:17is happening in real time where Jev's

5:57:19able to process these transactions and

5:57:21ask that question of hey is this fraud

5:57:23or not within fractions of a second.

5:57:26Now, we found that there's about four

5:57:27transactions that recently appeared that

5:57:30it thinks are fraud. Now, in parallel to

5:57:32that, we're able to kick off another

5:57:34part of our flow, which is doing more of

5:57:36a semantic analysis on the entire

5:57:38account. We're looking at that one

5:57:40transaction that came in for $48,000,

5:57:42which was flagged. And we're asking a

5:57:44question to Jev is, hey, is there enough

5:57:46information here for us to determine

5:57:48whether or not this is actually fraud?

5:57:50And just as we heard Dieago talk about

5:57:52this earlier where you have these new

5:57:54kind of paradigms and logic come up, we

5:57:56ended up uh finding from this that

5:57:58there's uh a scoring that took place and

5:58:01we needed more information. Um we were

5:58:03asking the question, do we have enough

5:58:04information for fraud? And for that what

5:58:06we're going to end up doing is kicking

5:58:07off a system 2 agent that's going to go

5:58:09spin up and run for much larger longer

5:58:12calling a bank API and finding that you

5:58:15know there was a $48,000 transaction

5:58:17from a payee. uh basically creating a

5:58:20new account on his phone and it was

5:58:22earlier that day that that account got

5:58:23created. So high indicator of fraud. We

5:58:26circle back and ask that same question

5:58:28now to Jev and ask hey do we have enough

5:58:30evidence now? Jev agrees and it finds

5:58:32100% sufficious activity and we would

5:58:35hand that off effectively to another

5:58:36grading model that combines all those

5:58:38transactions that we found with the

5:58:40system payload and then finds that you

5:58:42know there is fraud in this particular

5:58:43use case. um squared of you know one to

5:58:47five we got 4.97.

5:58:49Now stepping out of this um you know

5:58:52typically what would happen from this

5:58:53point onwards is that we'd go off to a

5:58:54human to review. We'd have somebody kind

5:58:56of review both the system one and system

5:58:58two logs and then find that there was

5:59:00you know potential fraud and freeze the

5:59:02account. But you know effectively what

5:59:04we're seeing from customers that are

5:59:05going through and implementing Jev

5:59:06inside of their workflows is that

5:59:08they're able to replace the typical LLM

5:59:11calls that are going on behind the

5:59:12scenes. And in in process, we're saving

5:59:14about 50 uh times uh we're we're

5:59:17increasing the speed about 50 times

5:59:20while also being 7.1 uh times cheaper um

5:59:23than calling a traditional LLM.

5:59:26Now, we we've gone through and and kind

5:59:28of shown how customers would use this

5:59:30within our technology, but we're also

5:59:32pointing the finger back inwards where

5:59:34we're trying to ask the question of how

5:59:36can UiPath use this within the products

5:59:38and services that we offer to be able to

5:59:40help customers out. And there's a number

5:59:42of different patterns that we're kind of

5:59:44exploring with Jev. Um, and today we're

5:59:46going to talk about um a re-ranking

5:59:48problem as well as uh what we're using

5:59:50inside of email mining. And so to set

5:59:53the stage for for kind of reranking um

5:59:55effectively what we're trying to do is

5:59:57run and add uh within our retrieval

6:00:00augmented generation within our agents

6:00:02the ability for us to find more

6:00:04appropriate chunks to be able to provide

6:00:06to our agent. And we do that by

6:00:08basically finding relevant documents

6:00:10first and then using um typically an LLM

6:00:13to do this. But we're experimenting with

6:00:15also using Jev to go through and find

6:00:17the various different chunks that make

6:00:19are most relevant to answering the

6:00:20question and then providing those as as

6:00:23context to the LLM. And we've run a

6:00:25couple exper experiments on this now.

6:00:27And I'd like to hand it back off to

6:00:28Chibi who's going to end up telling us a

6:00:30little bit more about what we were able

6:00:32to find.

6:00:33>> Yeah. So this is where things got really

6:00:35interesting for us. This is like the

6:00:37first week of Jev. We had got our hands

6:00:39on Je and we have these extremely strong

6:00:43benchmarks and uh eval sets that we had

6:00:46for reranking that we have been running

6:00:48for some time and we need those for like

6:00:50when we change from one model to

6:00:52another. uh and we realized okay why not

6:00:56take these eval sets and build a hill

6:00:59climbing harness and ask Astra to say

6:01:03okay this is my score I can get with

6:01:06NLLM here is Jeff here are the three

6:01:09APIs find a strategy which can achieve

6:01:12that score and let it run and it ran for

6:01:17the entire night for 14 hours in a

6:01:19session and it came up with so many

6:01:21different strategies where it worked,

6:01:23where it didn't work and then we were

6:01:25slowly seeing the number go up. Every

6:01:28two hours there will be one strategy

6:01:29would perform slightly better. This was

6:01:32so surprising for us because what we

6:01:34thought was Jeb as an API if you just

6:01:37call it like when you just send all of

6:01:41Yeah. So when we call it uh we were

6:01:44getting only 866 which was very very low

6:01:48at that point when we were trying it out

6:01:50when we were thinking how do we increase

6:01:51it and then it slowly got got to the

6:01:54number of 889 while it was uh trying out

6:01:58new strategies and that strategy is

6:02:00called tournament and I want to showcase

6:02:02that strategy which it which these hill

6:02:04climbing models actually come up with

6:02:06and how we were able to see the

6:02:08intelligence squeaked out of Jev just by

6:02:11applying JV in a different architecture

6:02:14and different call method. Uh and I want

6:02:16to take you through tournament now. So

6:02:19what tournament did is that it actually

6:02:21reimagined reanking differently. Instead

6:02:24of looking at every chunk and saying

6:02:26that it is how good relevant is it for

6:02:29my question, it created cohorts of

6:02:32different u different groups of these

6:02:35chunks and made them fight between each

6:02:37other to be the best most relevant chunk

6:02:40by sending thousands and thousands of

6:02:42Jews for each and every one of them and

6:02:44it found leaders in those particular

6:02:47cohorts and then the winners would be

6:02:48promoted to the next round and then in

6:02:51from those winners we would again find

6:02:53like top winners from those chunks and

6:02:55then we'll slowly increase and find the

6:02:58top most the cream will rise to the top

6:03:00and you'll find the uh the most relevant

6:03:02chunks on the whole list. So this worked

6:03:06extremely well and we were testing it

6:03:08and uh we were surprised to see that

6:03:10even with thousands of Jev calls and

6:03:13thou uh many millions of contexts being

6:03:16used uh we were still under a second we

6:03:19were able to make all of these calls

6:03:21right and uh this was a huge uh

6:03:23revelation for us and we realized that

6:03:26gel is a building block that we can

6:03:29adjust and we are we have to find these

6:03:32new strategies where we would be able to

6:03:34apply this building block and it might

6:03:37look less intelligent on its own but

6:03:39with these strategies and patterns we

6:03:42can get more intelligent with even the

6:03:44current job model that we have right and

6:03:47that's that gave us like 15x speed and

6:03:49we tested it out on uh our our live um

6:03:54indexes and saw how it performed and uh

6:03:56people were extremely surprised by this

6:03:58one and next we saw how uh Jio could be

6:04:03used um uh for mining. So mining is has

6:04:07a lot of stages. I'm going to talk only

6:04:09about the classification stage. And uh

6:04:11what happens is we have customers who

6:04:13have millions of emails in a support uh

6:04:17email box and uh what happens is they

6:04:19want to analyze and they want to dig

6:04:21into what happened in these emails so

6:04:22that they can take some actions. I'm

6:04:24sure every one of you guys have one of

6:04:25those email boxes. uh what happens is

6:04:28that like it's so hard to find the right

6:04:30categories and bucketizations without

6:04:32reading all of these emails. So what we

6:04:35used to do is that we would go and

6:04:37create uh fine-tuning to figure out a

6:04:40model that classifies them and it would

6:04:41take hours and maybe day to fine-tune

6:04:44the particular model for the

6:04:45classification story. But what we found

6:04:48was this is again through hill climbing

6:04:49we found this pattern where we use we

6:04:53use an LLM to find an initial candidate

6:04:56list of categories and then it would say

6:04:59that okay these are all the 15

6:05:00categories uh that I think the emails

6:05:02will fit then what we would do is that

6:05:05we'll use je and run all the million

6:05:07emails through it and see if where the

6:05:10buckets fill up and with that we can

6:05:12actually see which buckets are not

6:05:15filled properly which gets overfitted,

6:05:17which buckets are never filled out, and

6:05:19then we ask the LLM to go back and

6:05:22optimize those buckets again. And then

6:05:24once the buckets are rearranged, we'll

6:05:26do another classification of all the all

6:05:28the millions of emails one more time.

6:05:31Because Je is so cheap, we were able to

6:05:33do this multiple times and eventually

6:05:36you'll end up with a category list which

6:05:38is extremely fit towards the mailbox

6:05:40that you are dealing with because it's

6:05:42not the same across different different

6:05:43mailboxes. The categories are completely

6:05:45different. But the powerful thing is

6:05:47once you did all of this work with an

6:05:49LLM going through and finding these

6:05:51categories, you don't need the LM

6:05:53anymore because you have the categories

6:05:56which you where you have distilled down

6:05:58all the intelligence into these

6:06:00categories. Now you can just build a

6:06:02maestro flow which basically takes just

6:06:05the categories alone and uses gel. So

6:06:08whenever a new email comes, it only

6:06:10needs gel. It just looks at the uh looks

6:06:13at the email and fits the categories and

6:06:15we basically have a list of uh follow

6:06:18through. So if this matches this

6:06:20category, you are going to get an email.

6:06:21This team is going to get an email or

6:06:23that team is going to get an email so

6:06:25easily and we basically removed LM

6:06:26completely from the picture on this one.

6:06:29So and we saw a humongous speed increase

6:06:32on on just email processing. It was

6:06:34almost like at real time an email comes

6:06:36and suddenly somebody will get notified.

6:06:38So that was an amazing uh um amazing

6:06:42thing to see as well. So over the few

6:06:44months we were so excited to play around

6:06:46with Jev and every day the Slack threads

6:06:49would light up everywhere in the in the

6:06:51company that I found this pattern, I

6:06:52found that pattern and uh it was such an

6:06:55exciting time uh during these last three

6:06:57months and I wanted to share that there

6:07:00are so much more patterns and strategies

6:07:03to be found. We are only scratching the

6:07:06surface of this and I want us to

6:07:10try out and like share the strategies

6:07:11that we all find and collaborate here.

6:07:14>> And if you're interested in trying out

6:07:15UiPath Flow um we have it available for

6:07:18free right now. Scan the QR code on the

6:07:20screen here and we'll be able it'll take

6:07:21you into a portal where you can

6:07:22basically point your coding agent at it

6:07:24and get complete access to build your

6:07:26first automations there. Thank you for

6:07:27your time everyone.

6:07:29>> Thank you. [applause]