1
05:52:19,840 --> 05:52:22,788
I'm joined on stage by Kevin. Uh we're

2
05:52:22,798 --> 05:52:25,190
here to talk about how we believe that

3
05:52:25,200 --> 05:52:28,150
uh your automations are overthinking it.

4
05:52:28,160 --> 05:52:30,868
So um tell us a little bit about

5
05:52:30,878 --> 05:52:32,230
ourselves. We are from a company called

6
05:52:32,240 --> 05:52:35,110
UiPath. Uh we have been doing automation

7
05:52:35,120 --> 05:52:37,590
for the last 20 years. uh we've been

8
05:52:37,600 --> 05:52:40,390
doing uh RPA before and uh we service

9
05:52:40,400 --> 05:52:42,548
more than 10,000 organizations around

10
05:52:42,558 --> 05:52:45,750
the around the world. Um and we build uh

11
05:52:45,760 --> 05:52:48,708
an agentic process orchestration layer

12
05:52:48,718 --> 05:52:51,270
which basically brings agents, robots

13
05:52:51,280 --> 05:52:53,430
and people together to interact with

14
05:52:53,440 --> 05:52:55,830
each other and uh build extremely

15
05:52:55,840 --> 05:52:58,150
scalable uh automations that run for

16
05:52:58,160 --> 05:53:02,470
days and weeks. So we were inspired by

17
05:53:02,480 --> 05:53:04,470
this particular slide that you're seeing

18
05:53:04,480 --> 05:53:06,788
right now and you saw today today before

19
05:53:06,798 --> 05:53:10,150
us as well. Uh when uh Diego presented

20
05:53:10,160 --> 05:53:13,830
it 3 months ago the same slide uh it

21
05:53:13,840 --> 05:53:16,310
resonated really well with us because

22
05:53:16,320 --> 05:53:19,270
this is how we see automation as well.

23
05:53:19,280 --> 05:53:22,708
So the traditional software was is

24
05:53:22,718 --> 05:53:25,030
basically RPA. It's just robotic process

25
05:53:25,040 --> 05:53:27,270
automation if else just executes some

26
05:53:27,280 --> 05:53:28,628
task and then keeps going. And the

27
05:53:28,638 --> 05:53:30,470
control flow is deterministic. It just

28
05:53:30,480 --> 05:53:33,350
keeps going. And then when the LLMs came

29
05:53:33,360 --> 05:53:36,788
out, uh we all raced to go and build the

30
05:53:36,798 --> 05:53:38,628
just in time software where we have this

31
05:53:38,638 --> 05:53:40,868
one agent. We give it a bunch of tools.

32
05:53:40,878 --> 05:53:43,110
We give it a task and ask it to finish.

33
05:53:43,120 --> 05:53:45,510
And then we were seeing good results,

34
05:53:45,520 --> 05:53:47,110
bad results, and then we added

35
05:53:47,120 --> 05:53:49,190
evaluations to slowly build the results

36
05:53:49,200 --> 05:53:52,150
up. And then finally we landed on

37
05:53:52,160 --> 05:53:54,390
agentic process orchestration which kind

38
05:53:54,400 --> 05:53:57,350
of weaves in this cognitive decision

39
05:53:57,360 --> 05:54:01,030
through a logical uh decision layer. So

40
05:54:01,040 --> 05:54:04,470
you have deterministic calls and clear

41
05:54:04,480 --> 05:54:06,868
cognitive decision points where you

42
05:54:06,878 --> 05:54:09,190
would use an LLM to control the control

43
05:54:09,200 --> 05:54:11,590
flow rather than leaving it all in the

44
05:54:11,600 --> 05:54:15,590
hands of an agent. So at this time we we

45
05:54:15,600 --> 05:54:17,910
saw where J would fit in and when uh

46
05:54:17,920 --> 05:54:19,510
Diego was talking about Je we were

47
05:54:19,520 --> 05:54:22,788
really excited because we saw Je fit in

48
05:54:22,798 --> 05:54:26,708
in a very particular niche area. So u

49
05:54:26,718 --> 05:54:30,390
here we see that Je is a very powerful

50
05:54:30,400 --> 05:54:33,030
tool as Diego was saying it's a

51
05:54:33,040 --> 05:54:35,030
foundational tool that we would be able

52
05:54:35,040 --> 05:54:37,910
to integrate into all automation. Uh and

53
05:54:37,920 --> 05:54:40,310
it fit right in between a deterministic

54
05:54:40,320 --> 05:54:44,390
case and an agent case. So, Jev is

55
05:54:44,400 --> 05:54:46,628
extremely fast and most of you guys

56
05:54:46,638 --> 05:54:48,470
would have tried it and we were

57
05:54:48,480 --> 05:54:50,708
surprised by how fast and how parallel

58
05:54:50,718 --> 05:54:53,510
we could make it work and uh and how

59
05:54:53,520 --> 05:54:55,430
intelligent it was. So, we were able to

60
05:54:55,440 --> 05:54:57,350
benchmark it around how good it was

61
05:54:57,360 --> 05:54:59,830
compared to an LLM and was pretty good

62
05:54:59,840 --> 05:55:02,070
uh not as a full LM but it was pretty

63
05:55:02,080 --> 05:55:03,670
good at in terms of intelligence as

64
05:55:03,680 --> 05:55:06,310
well. So it falls in between and there

65
05:55:06,320 --> 05:55:09,110
are so many ways we could use je and uh

66
05:55:09,120 --> 05:55:11,190
I want to hand it off to Kevin to take

67
05:55:11,200 --> 05:55:13,350
you uh take you guys through a scenario

68
05:55:13,360 --> 05:55:16,390
where uh we use Jev in uh UiPath Maestro

69
05:55:16,400 --> 05:55:17,270
Flow.

70
05:55:17,280 --> 05:55:18,070
>> Yeah.

71
05:55:18,080 --> 05:55:21,190
>> Thanks Chibi. Um so hi everybody. Uh

72
05:55:21,200 --> 05:55:22,628
today I want to actually walk us through

73
05:55:22,638 --> 05:55:24,390
a little bit about what UiPath is

74
05:55:24,400 --> 05:55:26,788
building with UiPath Maestro Flow and

75
05:55:26,798 --> 05:55:29,110
talk through a little bit of how we

76
05:55:29,120 --> 05:55:31,510
would go through and coordinate humans,

77
05:55:31,520 --> 05:55:33,350
robots, and agents all together inside

78
05:55:33,360 --> 05:55:35,270
of one canvas. Uh what we're going to

79
05:55:35,280 --> 05:55:37,750
walk through today is essentially um you

80
05:55:37,760 --> 05:55:39,510
know essentially a flow that will

81
05:55:39,520 --> 05:55:41,110
demonstrate how we kind of weave all

82
05:55:41,120 --> 05:55:43,190
these things together. And the way that

83
05:55:43,200 --> 05:55:44,868
we're approaching this is that we're

84
05:55:44,878 --> 05:55:46,628
going to end up having system one agents

85
05:55:46,638 --> 05:55:48,310
which are going to be able to make quick

86
05:55:48,320 --> 05:55:50,868
intelligence happen within our process.

87
05:55:50,878 --> 05:55:52,548
Think about things like, hey, is this

88
05:55:52,558 --> 05:55:55,750
transaction fraud or not? Um,

89
05:55:55,760 --> 05:55:58,548
additionally, we'll have u system 2

90
05:55:58,558 --> 05:55:59,910
agents running where we're going to have

91
05:55:59,920 --> 05:56:01,830
more of methodical deeper investigation

92
05:56:01,840 --> 05:56:03,430
that's going on where agents are going

93
05:56:03,440 --> 05:56:05,110
off and calling tools. And we'll see

94
05:56:05,120 --> 05:56:06,628
kind of this pattern these patterns

95
05:56:06,638 --> 05:56:07,990
start to emerge where we're able to hand

96
05:56:08,000 --> 05:56:10,390
off from one to the next. And so with

97
05:56:10,400 --> 05:56:12,948
that, I'll I'll jump into an actual demo

98
05:56:12,958 --> 05:56:14,948
where we're going to demonstrate a

99
05:56:14,958 --> 05:56:18,230
account fraud uh scenario. And so our

100
05:56:18,240 --> 05:56:21,990
demo starts here with uh effectively a

101
05:56:22,000 --> 05:56:25,830
transaction coming in for $48,000.

102
05:56:25,840 --> 05:56:27,190
And this wasn't something that we were

103
05:56:27,200 --> 05:56:29,590
expecting. And so uh as I'm reviewing my

104
05:56:29,600 --> 05:56:30,788
bank portal here, I'm going to go ahead

105
05:56:30,798 --> 05:56:32,070
and report this as a fraudulent

106
05:56:32,080 --> 05:56:34,548
transaction. And so now this would end

107
05:56:34,558 --> 05:56:36,948
up smitting it off into our UiPath

108
05:56:36,958 --> 05:56:38,788
Maestro flow where this would act as

109
05:56:38,798 --> 05:56:40,868
essentially the trigger for this

110
05:56:40,878 --> 05:56:42,948
endto-end workflow to to go off and

111
05:56:42,958 --> 05:56:44,868
start processing. What we're looking at

112
05:56:44,878 --> 05:56:46,230
is a collection of various different

113
05:56:46,240 --> 05:56:49,190
nodes that are each going to kind of uh

114
05:56:49,200 --> 05:56:51,110
be orchestrated from the left side of

115
05:56:51,120 --> 05:56:53,110
the canvas over to the right. And we're

116
05:56:53,120 --> 05:56:54,948
going to use uh essentially a

117
05:56:54,958 --> 05:56:56,548
combination of system one and system two

118
05:56:56,558 --> 05:56:58,390
models to actually process this uh and

119
05:56:58,400 --> 05:57:01,110
do analysis of this is fraud or not. And

120
05:57:01,120 --> 05:57:02,628
so the first thing that actually takes

121
05:57:02,638 --> 05:57:04,548
place is we're going to go through and

122
05:57:04,558 --> 05:57:07,830
get account activity. We're going to

123
05:57:07,840 --> 05:57:09,990
send this off to Jev to basically pull

124
05:57:10,000 --> 05:57:12,390
60 transactions from the most recent uh

125
05:57:12,400 --> 05:57:14,310
credit card purchases that were made and

126
05:57:14,320 --> 05:57:16,070
analyze each one of those individually

127
05:57:16,080 --> 05:57:17,910
whether or not these are fraud. And this

128
05:57:17,920 --> 05:57:19,510
is happening in real time where Jev's

129
05:57:19,520 --> 05:57:21,430
able to process these transactions and

130
05:57:21,440 --> 05:57:23,350
ask that question of hey is this fraud

131
05:57:23,360 --> 05:57:26,150
or not within fractions of a second.

132
05:57:26,160 --> 05:57:27,750
Now, we found that there's about four

133
05:57:27,760 --> 05:57:29,990
transactions that recently appeared that

134
05:57:30,000 --> 05:57:32,788
it thinks are fraud. Now, in parallel to

135
05:57:32,798 --> 05:57:34,708
that, we're able to kick off another

136
05:57:34,718 --> 05:57:36,628
part of our flow, which is doing more of

137
05:57:36,638 --> 05:57:38,230
a semantic analysis on the entire

138
05:57:38,240 --> 05:57:40,070
account. We're looking at that one

139
05:57:40,080 --> 05:57:42,390
transaction that came in for $48,000,

140
05:57:42,400 --> 05:57:44,628
which was flagged. And we're asking a

141
05:57:44,638 --> 05:57:46,390
question to Jev is, hey, is there enough

142
05:57:46,400 --> 05:57:47,990
information here for us to determine

143
05:57:48,000 --> 05:57:50,070
whether or not this is actually fraud?

144
05:57:50,080 --> 05:57:52,150
And just as we heard Dieago talk about

145
05:57:52,160 --> 05:57:54,070
this earlier where you have these new

146
05:57:54,080 --> 05:57:56,708
kind of paradigms and logic come up, we

147
05:57:56,718 --> 05:57:58,628
ended up uh finding from this that

148
05:57:58,638 --> 05:58:01,110
there's uh a scoring that took place and

149
05:58:01,120 --> 05:58:03,030
we needed more information. Um we were

150
05:58:03,040 --> 05:58:04,150
asking the question, do we have enough

151
05:58:04,160 --> 05:58:06,548
information for fraud? And for that what

152
05:58:06,558 --> 05:58:07,830
we're going to end up doing is kicking

153
05:58:07,840 --> 05:58:09,750
off a system 2 agent that's going to go

154
05:58:09,760 --> 05:58:12,628
spin up and run for much larger longer

155
05:58:12,638 --> 05:58:15,590
calling a bank API and finding that you

156
05:58:15,600 --> 05:58:17,350
know there was a $48,000 transaction

157
05:58:17,360 --> 05:58:20,470
from a payee. uh basically creating a

158
05:58:20,480 --> 05:58:21,990
new account on his phone and it was

159
05:58:22,000 --> 05:58:23,830
earlier that day that that account got

160
05:58:23,840 --> 05:58:26,788
created. So high indicator of fraud. We

161
05:58:26,798 --> 05:58:28,470
circle back and ask that same question

162
05:58:28,480 --> 05:58:30,310
now to Jev and ask hey do we have enough

163
05:58:30,320 --> 05:58:32,548
evidence now? Jev agrees and it finds

164
05:58:32,558 --> 05:58:35,190
100% sufficious activity and we would

165
05:58:35,200 --> 05:58:36,948
hand that off effectively to another

166
05:58:36,958 --> 05:58:38,628
grading model that combines all those

167
05:58:38,638 --> 05:58:40,470
transactions that we found with the

168
05:58:40,480 --> 05:58:42,548
system payload and then finds that you

169
05:58:42,558 --> 05:58:43,830
know there is fraud in this particular

170
05:58:43,840 --> 05:58:47,030
use case. um squared of you know one to

171
05:58:47,040 --> 05:58:49,270
five we got 4.97.

172
05:58:49,280 --> 05:58:52,070
Now stepping out of this um you know

173
05:58:52,080 --> 05:58:53,270
typically what would happen from this

174
05:58:53,280 --> 05:58:54,628
point onwards is that we'd go off to a

175
05:58:54,638 --> 05:58:56,628
human to review. We'd have somebody kind

176
05:58:56,638 --> 05:58:58,150
of review both the system one and system

177
05:58:58,160 --> 05:59:00,548
two logs and then find that there was

178
05:59:00,558 --> 05:59:01,990
you know potential fraud and freeze the

179
05:59:02,000 --> 05:59:04,310
account. But you know effectively what

180
05:59:04,320 --> 05:59:05,510
we're seeing from customers that are

181
05:59:05,520 --> 05:59:06,948
going through and implementing Jev

182
05:59:06,958 --> 05:59:08,788
inside of their workflows is that

183
05:59:08,798 --> 05:59:11,190
they're able to replace the typical LLM

184
05:59:11,200 --> 05:59:12,470
calls that are going on behind the

185
05:59:12,480 --> 05:59:14,868
scenes. And in in process, we're saving

186
05:59:14,878 --> 05:59:17,910
about 50 uh times uh we're we're

187
05:59:17,920 --> 05:59:19,990
increasing the speed about 50 times

188
05:59:20,000 --> 05:59:23,590
while also being 7.1 uh times cheaper um

189
05:59:23,600 --> 05:59:26,548
than calling a traditional LLM.

190
05:59:26,558 --> 05:59:28,868
Now, we we've gone through and and kind

191
05:59:28,878 --> 05:59:30,548
of shown how customers would use this

192
05:59:30,558 --> 05:59:32,230
within our technology, but we're also

193
05:59:32,240 --> 05:59:34,070
pointing the finger back inwards where

194
05:59:34,080 --> 05:59:36,230
we're trying to ask the question of how

195
05:59:36,240 --> 05:59:38,390
can UiPath use this within the products

196
05:59:38,400 --> 05:59:40,628
and services that we offer to be able to

197
05:59:40,638 --> 05:59:42,948
help customers out. And there's a number

198
05:59:42,958 --> 05:59:44,230
of different patterns that we're kind of

199
05:59:44,240 --> 05:59:45,990
exploring with Jev. Um, and today we're

200
05:59:46,000 --> 05:59:48,150
going to talk about um a re-ranking

201
05:59:48,160 --> 05:59:50,470
problem as well as uh what we're using

202
05:59:50,480 --> 05:59:53,430
inside of email mining. And so to set

203
05:59:53,440 --> 05:59:55,830
the stage for for kind of reranking um

204
05:59:55,840 --> 05:59:57,670
effectively what we're trying to do is

205
05:59:57,680 --> 06:00:00,470
run and add uh within our retrieval

206
06:00:00,480 --> 06:00:02,390
augmented generation within our agents

207
06:00:02,400 --> 06:00:04,390
the ability for us to find more

208
06:00:04,400 --> 06:00:06,390
appropriate chunks to be able to provide

209
06:00:06,400 --> 06:00:08,470
to our agent. And we do that by

210
06:00:08,480 --> 06:00:10,548
basically finding relevant documents

211
06:00:10,558 --> 06:00:13,750
first and then using um typically an LLM

212
06:00:13,760 --> 06:00:15,190
to do this. But we're experimenting with

213
06:00:15,200 --> 06:00:17,350
also using Jev to go through and find

214
06:00:17,360 --> 06:00:19,270
the various different chunks that make

215
06:00:19,280 --> 06:00:20,708
are most relevant to answering the

216
06:00:20,718 --> 06:00:23,030
question and then providing those as as

217
06:00:23,040 --> 06:00:25,350
context to the LLM. And we've run a

218
06:00:25,360 --> 06:00:27,270
couple exper experiments on this now.

219
06:00:27,280 --> 06:00:28,548
And I'd like to hand it back off to

220
06:00:28,558 --> 06:00:30,310
Chibi who's going to end up telling us a

221
06:00:30,320 --> 06:00:32,070
little bit more about what we were able

222
06:00:32,080 --> 06:00:33,110
to find.

223
06:00:33,120 --> 06:00:35,830
>> Yeah. So this is where things got really

224
06:00:35,840 --> 06:00:37,350
interesting for us. This is like the

225
06:00:37,360 --> 06:00:39,510
first week of Jev. We had got our hands

226
06:00:39,520 --> 06:00:43,190
on Je and we have these extremely strong

227
06:00:43,200 --> 06:00:46,310
benchmarks and uh eval sets that we had

228
06:00:46,320 --> 06:00:48,150
for reranking that we have been running

229
06:00:48,160 --> 06:00:50,708
for some time and we need those for like

230
06:00:50,718 --> 06:00:52,548
when we change from one model to

231
06:00:52,558 --> 06:00:56,230
another. uh and we realized okay why not

232
06:00:56,240 --> 06:00:59,510
take these eval sets and build a hill

233
06:00:59,520 --> 06:01:03,750
climbing harness and ask Astra to say

234
06:01:03,760 --> 06:01:06,390
okay this is my score I can get with

235
06:01:06,400 --> 06:01:09,350
NLLM here is Jeff here are the three

236
06:01:09,360 --> 06:01:12,948
APIs find a strategy which can achieve

237
06:01:12,958 --> 06:01:17,190
that score and let it run and it ran for

238
06:01:17,200 --> 06:01:19,270
the entire night for 14 hours in a

239
06:01:19,280 --> 06:01:21,830
session and it came up with so many

240
06:01:21,840 --> 06:01:23,670
different strategies where it worked,

241
06:01:23,680 --> 06:01:25,670
where it didn't work and then we were

242
06:01:25,680 --> 06:01:28,150
slowly seeing the number go up. Every

243
06:01:28,160 --> 06:01:29,670
two hours there will be one strategy

244
06:01:29,680 --> 06:01:32,310
would perform slightly better. This was

245
06:01:32,320 --> 06:01:34,390
so surprising for us because what we

246
06:01:34,400 --> 06:01:37,670
thought was Jeb as an API if you just

247
06:01:37,680 --> 06:01:41,830
call it like when you just send all of

248
06:01:41,840 --> 06:01:44,310
Yeah. So when we call it uh we were

249
06:01:44,320 --> 06:01:48,070
getting only 866 which was very very low

250
06:01:48,080 --> 06:01:50,150
at that point when we were trying it out

251
06:01:50,160 --> 06:01:51,910
when we were thinking how do we increase

252
06:01:51,920 --> 06:01:54,628
it and then it slowly got got to the

253
06:01:54,638 --> 06:01:58,070
number of 889 while it was uh trying out

254
06:01:58,080 --> 06:01:59,990
new strategies and that strategy is

255
06:02:00,000 --> 06:02:01,990
called tournament and I want to showcase

256
06:02:02,000 --> 06:02:04,708
that strategy which it which these hill

257
06:02:04,718 --> 06:02:06,708
climbing models actually come up with

258
06:02:06,718 --> 06:02:08,390
and how we were able to see the

259
06:02:08,400 --> 06:02:11,430
intelligence squeaked out of Jev just by

260
06:02:11,440 --> 06:02:13,990
applying JV in a different architecture

261
06:02:14,000 --> 06:02:16,708
and different call method. Uh and I want

262
06:02:16,718 --> 06:02:19,590
to take you through tournament now. So

263
06:02:19,600 --> 06:02:21,670
what tournament did is that it actually

264
06:02:21,680 --> 06:02:24,390
reimagined reanking differently. Instead

265
06:02:24,400 --> 06:02:26,948
of looking at every chunk and saying

266
06:02:26,958 --> 06:02:29,350
that it is how good relevant is it for

267
06:02:29,360 --> 06:02:32,788
my question, it created cohorts of

268
06:02:32,798 --> 06:02:35,510
different u different groups of these

269
06:02:35,520 --> 06:02:37,590
chunks and made them fight between each

270
06:02:37,600 --> 06:02:40,150
other to be the best most relevant chunk

271
06:02:40,160 --> 06:02:42,230
by sending thousands and thousands of

272
06:02:42,240 --> 06:02:44,548
Jews for each and every one of them and

273
06:02:44,558 --> 06:02:47,030
it found leaders in those particular

274
06:02:47,040 --> 06:02:48,868
cohorts and then the winners would be

275
06:02:48,878 --> 06:02:51,190
promoted to the next round and then in

276
06:02:51,200 --> 06:02:53,350
from those winners we would again find

277
06:02:53,360 --> 06:02:55,590
like top winners from those chunks and

278
06:02:55,600 --> 06:02:58,230
then we'll slowly increase and find the

279
06:02:58,240 --> 06:03:00,470
top most the cream will rise to the top

280
06:03:00,480 --> 06:03:02,788
and you'll find the uh the most relevant

281
06:03:02,798 --> 06:03:06,230
chunks on the whole list. So this worked

282
06:03:06,240 --> 06:03:08,070
extremely well and we were testing it

283
06:03:08,080 --> 06:03:10,310
and uh we were surprised to see that

284
06:03:10,320 --> 06:03:13,430
even with thousands of Jev calls and

285
06:03:13,440 --> 06:03:16,708
thou uh many millions of contexts being

286
06:03:16,718 --> 06:03:19,830
used uh we were still under a second we

287
06:03:19,840 --> 06:03:21,590
were able to make all of these calls

288
06:03:21,600 --> 06:03:23,670
right and uh this was a huge uh

289
06:03:23,680 --> 06:03:26,150
revelation for us and we realized that

290
06:03:26,160 --> 06:03:29,270
gel is a building block that we can

291
06:03:29,280 --> 06:03:32,150
adjust and we are we have to find these

292
06:03:32,160 --> 06:03:34,470
new strategies where we would be able to

293
06:03:34,480 --> 06:03:37,270
apply this building block and it might

294
06:03:37,280 --> 06:03:39,510
look less intelligent on its own but

295
06:03:39,520 --> 06:03:42,470
with these strategies and patterns we

296
06:03:42,480 --> 06:03:44,390
can get more intelligent with even the

297
06:03:44,400 --> 06:03:47,030
current job model that we have right and

298
06:03:47,040 --> 06:03:49,750
that's that gave us like 15x speed and

299
06:03:49,760 --> 06:03:54,230
we tested it out on uh our our live um

300
06:03:54,240 --> 06:03:56,708
indexes and saw how it performed and uh

301
06:03:56,718 --> 06:03:58,390
people were extremely surprised by this

302
06:03:58,400 --> 06:04:03,110
one and next we saw how uh Jio could be

303
06:04:03,120 --> 06:04:07,430
used um uh for mining. So mining is has

304
06:04:07,440 --> 06:04:09,110
a lot of stages. I'm going to talk only

305
06:04:09,120 --> 06:04:11,590
about the classification stage. And uh

306
06:04:11,600 --> 06:04:13,350
what happens is we have customers who

307
06:04:13,360 --> 06:04:17,270
have millions of emails in a support uh

308
06:04:17,280 --> 06:04:19,750
email box and uh what happens is they

309
06:04:19,760 --> 06:04:21,030
want to analyze and they want to dig

310
06:04:21,040 --> 06:04:22,628
into what happened in these emails so

311
06:04:22,638 --> 06:04:24,390
that they can take some actions. I'm

312
06:04:24,400 --> 06:04:25,830
sure every one of you guys have one of

313
06:04:25,840 --> 06:04:28,788
those email boxes. uh what happens is

314
06:04:28,798 --> 06:04:30,948
that like it's so hard to find the right

315
06:04:30,958 --> 06:04:32,948
categories and bucketizations without

316
06:04:32,958 --> 06:04:35,830
reading all of these emails. So what we

317
06:04:35,840 --> 06:04:37,510
used to do is that we would go and

318
06:04:37,520 --> 06:04:39,990
create uh fine-tuning to figure out a

319
06:04:40,000 --> 06:04:41,750
model that classifies them and it would

320
06:04:41,760 --> 06:04:44,470
take hours and maybe day to fine-tune

321
06:04:44,480 --> 06:04:45,670
the particular model for the

322
06:04:45,680 --> 06:04:48,070
classification story. But what we found

323
06:04:48,080 --> 06:04:49,830
was this is again through hill climbing

324
06:04:49,840 --> 06:04:53,110
we found this pattern where we use we

325
06:04:53,120 --> 06:04:56,470
use an LLM to find an initial candidate

326
06:04:56,480 --> 06:04:59,590
list of categories and then it would say

327
06:04:59,600 --> 06:05:00,868
that okay these are all the 15

328
06:05:00,878 --> 06:05:02,948
categories uh that I think the emails

329
06:05:02,958 --> 06:05:05,030
will fit then what we would do is that

330
06:05:05,040 --> 06:05:07,510
we'll use je and run all the million

331
06:05:07,520 --> 06:05:09,990
emails through it and see if where the

332
06:05:10,000 --> 06:05:12,948
buckets fill up and with that we can

333
06:05:12,958 --> 06:05:15,350
actually see which buckets are not

334
06:05:15,360 --> 06:05:17,830
filled properly which gets overfitted,

335
06:05:17,840 --> 06:05:19,750
which buckets are never filled out, and

336
06:05:19,760 --> 06:05:21,990
then we ask the LLM to go back and

337
06:05:22,000 --> 06:05:24,150
optimize those buckets again. And then

338
06:05:24,160 --> 06:05:26,470
once the buckets are rearranged, we'll

339
06:05:26,480 --> 06:05:28,788
do another classification of all the all

340
06:05:28,798 --> 06:05:31,350
the millions of emails one more time.

341
06:05:31,360 --> 06:05:33,350
Because Je is so cheap, we were able to

342
06:05:33,360 --> 06:05:36,150
do this multiple times and eventually

343
06:05:36,160 --> 06:05:38,470
you'll end up with a category list which

344
06:05:38,480 --> 06:05:40,628
is extremely fit towards the mailbox

345
06:05:40,638 --> 06:05:41,990
that you are dealing with because it's

346
06:05:42,000 --> 06:05:43,910
not the same across different different

347
06:05:43,920 --> 06:05:45,590
mailboxes. The categories are completely

348
06:05:45,600 --> 06:05:47,750
different. But the powerful thing is

349
06:05:47,760 --> 06:05:49,750
once you did all of this work with an

350
06:05:49,760 --> 06:05:51,590
LLM going through and finding these

351
06:05:51,600 --> 06:05:53,670
categories, you don't need the LM

352
06:05:53,680 --> 06:05:56,788
anymore because you have the categories

353
06:05:56,798 --> 06:05:58,788
which you where you have distilled down

354
06:05:58,798 --> 06:05:59,990
all the intelligence into these

355
06:06:00,000 --> 06:06:02,310
categories. Now you can just build a

356
06:06:02,320 --> 06:06:05,430
maestro flow which basically takes just

357
06:06:05,440 --> 06:06:08,310
the categories alone and uses gel. So

358
06:06:08,320 --> 06:06:10,470
whenever a new email comes, it only

359
06:06:10,480 --> 06:06:13,030
needs gel. It just looks at the uh looks

360
06:06:13,040 --> 06:06:15,590
at the email and fits the categories and

361
06:06:15,600 --> 06:06:18,470
we basically have a list of uh follow

362
06:06:18,480 --> 06:06:20,070
through. So if this matches this

363
06:06:20,080 --> 06:06:21,750
category, you are going to get an email.

364
06:06:21,760 --> 06:06:23,110
This team is going to get an email or

365
06:06:23,120 --> 06:06:25,030
that team is going to get an email so

366
06:06:25,040 --> 06:06:26,948
easily and we basically removed LM

367
06:06:26,958 --> 06:06:29,190
completely from the picture on this one.

368
06:06:29,200 --> 06:06:32,628
So and we saw a humongous speed increase

369
06:06:32,638 --> 06:06:34,708
on on just email processing. It was

370
06:06:34,718 --> 06:06:36,708
almost like at real time an email comes

371
06:06:36,718 --> 06:06:38,628
and suddenly somebody will get notified.

372
06:06:38,638 --> 06:06:41,990
So that was an amazing uh um amazing

373
06:06:42,000 --> 06:06:44,150
thing to see as well. So over the few

374
06:06:44,160 --> 06:06:46,868
months we were so excited to play around

375
06:06:46,878 --> 06:06:49,350
with Jev and every day the Slack threads

376
06:06:49,360 --> 06:06:51,190
would light up everywhere in the in the

377
06:06:51,200 --> 06:06:52,788
company that I found this pattern, I

378
06:06:52,798 --> 06:06:55,430
found that pattern and uh it was such an

379
06:06:55,440 --> 06:06:57,670
exciting time uh during these last three

380
06:06:57,680 --> 06:07:00,788
months and I wanted to share that there

381
06:07:00,798 --> 06:07:03,110
are so much more patterns and strategies

382
06:07:03,120 --> 06:07:06,070
to be found. We are only scratching the

383
06:07:06,080 --> 06:07:10,070
surface of this and I want us to

384
06:07:10,080 --> 06:07:11,910
try out and like share the strategies

385
06:07:11,920 --> 06:07:14,310
that we all find and collaborate here.

386
06:07:14,320 --> 06:07:15,670
>> And if you're interested in trying out

387
06:07:15,680 --> 06:07:18,150
UiPath Flow um we have it available for

388
06:07:18,160 --> 06:07:20,070
free right now. Scan the QR code on the

389
06:07:20,080 --> 06:07:21,750
screen here and we'll be able it'll take

390
06:07:21,760 --> 06:07:22,868
you into a portal where you can

391
06:07:22,878 --> 06:07:24,470
basically point your coding agent at it

392
06:07:24,480 --> 06:07:25,990
and get complete access to build your

393
06:07:26,000 --> 06:07:27,670
first automations there. Thank you for

394
06:07:27,680 --> 06:07:29,030
your time everyone.

395
06:07:29,040 --> 06:07:31,000
>> Thank you. [applause]