# Your Automation Is Overthinking It Source: https://www.youtube.com/watch?v=ZZ19zhYnDKY Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session. [05:52:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21139s) I'm joined on stage by Kevin. Uh we're [05:52:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21142s) here to talk about how we believe that [05:52:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21145s) uh your automations are overthinking it. [05:52:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21148s) So um tell us a little bit about [05:52:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21150s) ourselves. We are from a company called [05:52:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21152s) UiPath. Uh we have been doing automation [05:52:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21155s) for the last 20 years. uh we've been [05:52:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21157s) doing uh RPA before and uh we service [05:52:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21160s) more than 10,000 organizations around [05:52:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21162s) the around the world. Um and we build uh [05:52:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21165s) an agentic process orchestration layer [05:52:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21168s) which basically brings agents, robots [05:52:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21171s) and people together to interact with [05:52:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21173s) each other and uh build extremely [05:52:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21175s) scalable uh automations that run for [05:52:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21178s) days and weeks. So we were inspired by [05:53:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21182s) this particular slide that you're seeing [05:53:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21184s) right now and you saw today today before [05:53:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21186s) us as well. Uh when uh Diego presented [05:53:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21190s) it 3 months ago the same slide uh it [05:53:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21193s) resonated really well with us because [05:53:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21196s) this is how we see automation as well. [05:53:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21199s) So the traditional software was is [05:53:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21202s) basically RPA. It's just robotic process [05:53:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21205s) automation if else just executes some [05:53:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21207s) task and then keeps going. And the [05:53:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21208s) control flow is deterministic. It just [05:53:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21210s) keeps going. And then when the LLMs came [05:53:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21213s) out, uh we all raced to go and build the [05:53:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21216s) just in time software where we have this [05:53:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21218s) one agent. We give it a bunch of tools. [05:53:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21220s) We give it a task and ask it to finish. [05:53:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21223s) And then we were seeing good results, [05:53:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21225s) bad results, and then we added [05:53:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21227s) evaluations to slowly build the results [05:53:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21229s) up. And then finally we landed on [05:53:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21232s) agentic process orchestration which kind [05:53:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21234s) of weaves in this cognitive decision [05:53:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21237s) through a logical uh decision layer. So [05:54:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21241s) you have deterministic calls and clear [05:54:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21244s) cognitive decision points where you [05:54:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21246s) would use an LLM to control the control [05:54:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21249s) flow rather than leaving it all in the [05:54:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21251s) hands of an agent. So at this time we we [05:54:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21255s) saw where J would fit in and when uh [05:54:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21257s) Diego was talking about Je we were [05:54:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21259s) really excited because we saw Je fit in [05:54:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21262s) in a very particular niche area. So u [05:54:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21266s) here we see that Je is a very powerful [05:54:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21270s) tool as Diego was saying it's a [05:54:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21273s) foundational tool that we would be able [05:54:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21275s) to integrate into all automation. Uh and [05:54:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21277s) it fit right in between a deterministic [05:54:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21280s) case and an agent case. So, Jev is [05:54:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21284s) extremely fast and most of you guys [05:54:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21286s) would have tried it and we were [05:54:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21288s) surprised by how fast and how parallel [05:54:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21290s) we could make it work and uh and how [05:54:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21293s) intelligent it was. So, we were able to [05:54:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21295s) benchmark it around how good it was [05:54:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21297s) compared to an LLM and was pretty good [05:54:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21299s) uh not as a full LM but it was pretty [05:55:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21302s) good at in terms of intelligence as [05:55:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21303s) well. So it falls in between and there [05:55:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21306s) are so many ways we could use je and uh [05:55:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21309s) I want to hand it off to Kevin to take [05:55:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21311s) you uh take you guys through a scenario [05:55:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21313s) where uh we use Jev in uh UiPath Maestro [05:55:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21316s) Flow. [05:55:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21317s) >> Yeah. [05:55:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21318s) >> Thanks Chibi. Um so hi everybody. Uh [05:55:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21321s) today I want to actually walk us through [05:55:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21322s) a little bit about what UiPath is [05:55:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21324s) building with UiPath Maestro Flow and [05:55:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21326s) talk through a little bit of how we [05:55:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21329s) would go through and coordinate humans, [05:55:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21331s) robots, and agents all together inside [05:55:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21333s) of one canvas. Uh what we're going to [05:55:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21335s) walk through today is essentially um you [05:55:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21337s) know essentially a flow that will [05:55:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21339s) demonstrate how we kind of weave all [05:55:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21341s) these things together. And the way that [05:55:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21343s) we're approaching this is that we're [05:55:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21344s) going to end up having system one agents [05:55:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21346s) which are going to be able to make quick [05:55:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21348s) intelligence happen within our process. [05:55:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21350s) Think about things like, hey, is this [05:55:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21352s) transaction fraud or not? Um, [05:55:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21355s) additionally, we'll have u system 2 [05:55:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21358s) agents running where we're going to have [05:55:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21359s) more of methodical deeper investigation [05:56:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21361s) that's going on where agents are going [05:56:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21363s) off and calling tools. And we'll see [05:56:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21365s) kind of this pattern these patterns [05:56:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21366s) start to emerge where we're able to hand [05:56:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21368s) off from one to the next. And so with [05:56:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21370s) that, I'll I'll jump into an actual demo [05:56:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21372s) where we're going to demonstrate a [05:56:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21374s) account fraud uh scenario. And so our [05:56:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21378s) demo starts here with uh effectively a [05:56:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21382s) transaction coming in for $48,000. [05:56:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21385s) And this wasn't something that we were [05:56:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21387s) expecting. And so uh as I'm reviewing my [05:56:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21389s) bank portal here, I'm going to go ahead [05:56:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21390s) and report this as a fraudulent [05:56:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21392s) transaction. And so now this would end [05:56:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21394s) up smitting it off into our UiPath [05:56:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21396s) Maestro flow where this would act as [05:56:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21398s) essentially the trigger for this [05:56:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21400s) endto-end workflow to to go off and [05:56:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21402s) start processing. What we're looking at [05:56:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21404s) is a collection of various different [05:56:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21406s) nodes that are each going to kind of uh [05:56:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21409s) be orchestrated from the left side of [05:56:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21411s) the canvas over to the right. And we're [05:56:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21413s) going to use uh essentially a [05:56:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21414s) combination of system one and system two [05:56:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21416s) models to actually process this uh and [05:56:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21418s) do analysis of this is fraud or not. And [05:57:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21421s) so the first thing that actually takes [05:57:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21422s) place is we're going to go through and [05:57:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21424s) get account activity. We're going to [05:57:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21427s) send this off to Jev to basically pull [05:57:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21430s) 60 transactions from the most recent uh [05:57:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21432s) credit card purchases that were made and [05:57:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21434s) analyze each one of those individually [05:57:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21436s) whether or not these are fraud. And this [05:57:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21437s) is happening in real time where Jev's [05:57:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21439s) able to process these transactions and [05:57:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21441s) ask that question of hey is this fraud [05:57:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21443s) or not within fractions of a second. [05:57:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21446s) Now, we found that there's about four [05:57:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21447s) transactions that recently appeared that [05:57:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21450s) it thinks are fraud. Now, in parallel to [05:57:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21452s) that, we're able to kick off another [05:57:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21454s) part of our flow, which is doing more of [05:57:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21456s) a semantic analysis on the entire [05:57:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21458s) account. We're looking at that one [05:57:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21460s) transaction that came in for $48,000, [05:57:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21462s) which was flagged. And we're asking a [05:57:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21464s) question to Jev is, hey, is there enough [05:57:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21466s) information here for us to determine [05:57:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21468s) whether or not this is actually fraud? [05:57:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21470s) And just as we heard Dieago talk about [05:57:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21472s) this earlier where you have these new [05:57:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21474s) kind of paradigms and logic come up, we [05:57:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21476s) ended up uh finding from this that [05:57:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21478s) there's uh a scoring that took place and [05:58:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21481s) we needed more information. Um we were [05:58:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21483s) asking the question, do we have enough [05:58:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21484s) information for fraud? And for that what [05:58:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21486s) we're going to end up doing is kicking [05:58:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21487s) off a system 2 agent that's going to go [05:58:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21489s) spin up and run for much larger longer [05:58:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21492s) calling a bank API and finding that you [05:58:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21495s) know there was a $48,000 transaction [05:58:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21497s) from a payee. uh basically creating a [05:58:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21500s) new account on his phone and it was [05:58:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21502s) earlier that day that that account got [05:58:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21503s) created. So high indicator of fraud. We [05:58:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21506s) circle back and ask that same question [05:58:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21508s) now to Jev and ask hey do we have enough [05:58:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21510s) evidence now? Jev agrees and it finds [05:58:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21512s) 100% sufficious activity and we would [05:58:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21515s) hand that off effectively to another [05:58:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21516s) grading model that combines all those [05:58:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21518s) transactions that we found with the [05:58:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21520s) system payload and then finds that you [05:58:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21522s) know there is fraud in this particular [05:58:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21523s) use case. um squared of you know one to [05:58:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21527s) five we got 4.97. [05:58:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21529s) Now stepping out of this um you know [05:58:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21532s) typically what would happen from this [05:58:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21533s) point onwards is that we'd go off to a [05:58:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21534s) human to review. We'd have somebody kind [05:58:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21536s) of review both the system one and system [05:58:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21538s) two logs and then find that there was [05:59:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21540s) you know potential fraud and freeze the [05:59:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21542s) account. But you know effectively what [05:59:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21544s) we're seeing from customers that are [05:59:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21545s) going through and implementing Jev [05:59:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21546s) inside of their workflows is that [05:59:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21548s) they're able to replace the typical LLM [05:59:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21551s) calls that are going on behind the [05:59:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21552s) scenes. And in in process, we're saving [05:59:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21554s) about 50 uh times uh we're we're [05:59:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21557s) increasing the speed about 50 times [05:59:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21560s) while also being 7.1 uh times cheaper um [05:59:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21563s) than calling a traditional LLM. [05:59:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21566s) Now, we we've gone through and and kind [05:59:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21568s) of shown how customers would use this [05:59:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21570s) within our technology, but we're also [05:59:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21572s) pointing the finger back inwards where [05:59:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21574s) we're trying to ask the question of how [05:59:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21576s) can UiPath use this within the products [05:59:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21578s) and services that we offer to be able to [05:59:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21580s) help customers out. And there's a number [05:59:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21582s) of different patterns that we're kind of [05:59:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21584s) exploring with Jev. Um, and today we're [05:59:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21586s) going to talk about um a re-ranking [05:59:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21588s) problem as well as uh what we're using [05:59:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21590s) inside of email mining. And so to set [05:59:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21593s) the stage for for kind of reranking um [05:59:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21595s) effectively what we're trying to do is [05:59:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21597s) run and add uh within our retrieval [06:00:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21600s) augmented generation within our agents [06:00:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21602s) the ability for us to find more [06:00:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21604s) appropriate chunks to be able to provide [06:00:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21606s) to our agent. And we do that by [06:00:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21608s) basically finding relevant documents [06:00:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21610s) first and then using um typically an LLM [06:00:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21613s) to do this. But we're experimenting with [06:00:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21615s) also using Jev to go through and find [06:00:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21617s) the various different chunks that make [06:00:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21619s) are most relevant to answering the [06:00:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21620s) question and then providing those as as [06:00:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21623s) context to the LLM. And we've run a [06:00:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21625s) couple exper experiments on this now. [06:00:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21627s) And I'd like to hand it back off to [06:00:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21628s) Chibi who's going to end up telling us a [06:00:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21630s) little bit more about what we were able [06:00:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21632s) to find. [06:00:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21633s) >> Yeah. So this is where things got really [06:00:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21635s) interesting for us. This is like the [06:00:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21637s) first week of Jev. We had got our hands [06:00:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21639s) on Je and we have these extremely strong [06:00:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21643s) benchmarks and uh eval sets that we had [06:00:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21646s) for reranking that we have been running [06:00:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21648s) for some time and we need those for like [06:00:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21650s) when we change from one model to [06:00:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21652s) another. uh and we realized okay why not [06:00:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21656s) take these eval sets and build a hill [06:00:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21659s) climbing harness and ask Astra to say [06:01:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21663s) okay this is my score I can get with [06:01:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21666s) NLLM here is Jeff here are the three [06:01:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21669s) APIs find a strategy which can achieve [06:01:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21672s) that score and let it run and it ran for [06:01:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21677s) the entire night for 14 hours in a [06:01:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21679s) session and it came up with so many [06:01:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21681s) different strategies where it worked, [06:01:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21683s) where it didn't work and then we were [06:01:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21685s) slowly seeing the number go up. Every [06:01:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21688s) two hours there will be one strategy [06:01:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21689s) would perform slightly better. This was [06:01:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21692s) so surprising for us because what we [06:01:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21694s) thought was Jeb as an API if you just [06:01:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21697s) call it like when you just send all of [06:01:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21701s) Yeah. So when we call it uh we were [06:01:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21704s) getting only 866 which was very very low [06:01:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21708s) at that point when we were trying it out [06:01:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21710s) when we were thinking how do we increase [06:01:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21711s) it and then it slowly got got to the [06:01:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21714s) number of 889 while it was uh trying out [06:01:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21718s) new strategies and that strategy is [06:02:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21720s) called tournament and I want to showcase [06:02:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21722s) that strategy which it which these hill [06:02:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21724s) climbing models actually come up with [06:02:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21726s) and how we were able to see the [06:02:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21728s) intelligence squeaked out of Jev just by [06:02:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21731s) applying JV in a different architecture [06:02:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21734s) and different call method. Uh and I want [06:02:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21736s) to take you through tournament now. So [06:02:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21739s) what tournament did is that it actually [06:02:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21741s) reimagined reanking differently. Instead [06:02:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21744s) of looking at every chunk and saying [06:02:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21746s) that it is how good relevant is it for [06:02:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21749s) my question, it created cohorts of [06:02:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21752s) different u different groups of these [06:02:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21755s) chunks and made them fight between each [06:02:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21757s) other to be the best most relevant chunk [06:02:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21760s) by sending thousands and thousands of [06:02:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21762s) Jews for each and every one of them and [06:02:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21764s) it found leaders in those particular [06:02:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21767s) cohorts and then the winners would be [06:02:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21768s) promoted to the next round and then in [06:02:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21771s) from those winners we would again find [06:02:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21773s) like top winners from those chunks and [06:02:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21775s) then we'll slowly increase and find the [06:02:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21778s) top most the cream will rise to the top [06:03:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21780s) and you'll find the uh the most relevant [06:03:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21782s) chunks on the whole list. So this worked [06:03:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21786s) extremely well and we were testing it [06:03:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21788s) and uh we were surprised to see that [06:03:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21790s) even with thousands of Jev calls and [06:03:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21793s) thou uh many millions of contexts being [06:03:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21796s) used uh we were still under a second we [06:03:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21799s) were able to make all of these calls [06:03:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21801s) right and uh this was a huge uh [06:03:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21803s) revelation for us and we realized that [06:03:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21806s) gel is a building block that we can [06:03:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21809s) adjust and we are we have to find these [06:03:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21812s) new strategies where we would be able to [06:03:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21814s) apply this building block and it might [06:03:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21817s) look less intelligent on its own but [06:03:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21819s) with these strategies and patterns we [06:03:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21822s) can get more intelligent with even the [06:03:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21824s) current job model that we have right and [06:03:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21827s) that's that gave us like 15x speed and [06:03:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21829s) we tested it out on uh our our live um [06:03:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21834s) indexes and saw how it performed and uh [06:03:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21836s) people were extremely surprised by this [06:03:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21838s) one and next we saw how uh Jio could be [06:04:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21843s) used um uh for mining. So mining is has [06:04:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21847s) a lot of stages. I'm going to talk only [06:04:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21849s) about the classification stage. And uh [06:04:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21851s) what happens is we have customers who [06:04:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21853s) have millions of emails in a support uh [06:04:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21857s) email box and uh what happens is they [06:04:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21859s) want to analyze and they want to dig [06:04:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21861s) into what happened in these emails so [06:04:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21862s) that they can take some actions. I'm [06:04:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21864s) sure every one of you guys have one of [06:04:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21865s) those email boxes. uh what happens is [06:04:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21868s) that like it's so hard to find the right [06:04:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21870s) categories and bucketizations without [06:04:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21872s) reading all of these emails. So what we [06:04:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21875s) used to do is that we would go and [06:04:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21877s) create uh fine-tuning to figure out a [06:04:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21880s) model that classifies them and it would [06:04:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21881s) take hours and maybe day to fine-tune [06:04:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21884s) the particular model for the [06:04:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21885s) classification story. But what we found [06:04:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21888s) was this is again through hill climbing [06:04:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21889s) we found this pattern where we use we [06:04:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21893s) use an LLM to find an initial candidate [06:04:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21896s) list of categories and then it would say [06:04:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21899s) that okay these are all the 15 [06:05:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21900s) categories uh that I think the emails [06:05:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21902s) will fit then what we would do is that [06:05:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21905s) we'll use je and run all the million [06:05:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21907s) emails through it and see if where the [06:05:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21910s) buckets fill up and with that we can [06:05:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21912s) actually see which buckets are not [06:05:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21915s) filled properly which gets overfitted, [06:05:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21917s) which buckets are never filled out, and [06:05:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21919s) then we ask the LLM to go back and [06:05:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21922s) optimize those buckets again. And then [06:05:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21924s) once the buckets are rearranged, we'll [06:05:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21926s) do another classification of all the all [06:05:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21928s) the millions of emails one more time. [06:05:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21931s) Because Je is so cheap, we were able to [06:05:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21933s) do this multiple times and eventually [06:05:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21936s) you'll end up with a category list which [06:05:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21938s) is extremely fit towards the mailbox [06:05:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21940s) that you are dealing with because it's [06:05:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21942s) not the same across different different [06:05:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21943s) mailboxes. The categories are completely [06:05:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21945s) different. But the powerful thing is [06:05:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21947s) once you did all of this work with an [06:05:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21949s) LLM going through and finding these [06:05:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21951s) categories, you don't need the LM [06:05:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21953s) anymore because you have the categories [06:05:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21956s) which you where you have distilled down [06:05:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21958s) all the intelligence into these [06:06:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21960s) categories. Now you can just build a [06:06:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21962s) maestro flow which basically takes just [06:06:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21965s) the categories alone and uses gel. So [06:06:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21968s) whenever a new email comes, it only [06:06:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21970s) needs gel. It just looks at the uh looks [06:06:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21973s) at the email and fits the categories and [06:06:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21975s) we basically have a list of uh follow [06:06:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21978s) through. So if this matches this [06:06:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21980s) category, you are going to get an email. [06:06:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21981s) This team is going to get an email or [06:06:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21983s) that team is going to get an email so [06:06:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21985s) easily and we basically removed LM [06:06:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21986s) completely from the picture on this one. [06:06:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21989s) So and we saw a humongous speed increase [06:06:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21992s) on on just email processing. It was [06:06:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21994s) almost like at real time an email comes [06:06:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21996s) and suddenly somebody will get notified. [06:06:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21998s) So that was an amazing uh um amazing [06:06:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22002s) thing to see as well. So over the few [06:06:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22004s) months we were so excited to play around [06:06:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22006s) with Jev and every day the Slack threads [06:06:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22009s) would light up everywhere in the in the [06:06:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22011s) company that I found this pattern, I [06:06:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22012s) found that pattern and uh it was such an [06:06:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22015s) exciting time uh during these last three [06:06:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22017s) months and I wanted to share that there [06:07:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22020s) are so much more patterns and strategies [06:07:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22023s) to be found. We are only scratching the [06:07:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22026s) surface of this and I want us to [06:07:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22030s) try out and like share the strategies [06:07:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22031s) that we all find and collaborate here. [06:07:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22034s) >> And if you're interested in trying out [06:07:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22035s) UiPath Flow um we have it available for [06:07:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22038s) free right now. Scan the QR code on the [06:07:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22040s) screen here and we'll be able it'll take [06:07:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22041s) you into a portal where you can [06:07:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22042s) basically point your coding agent at it [06:07:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22044s) and get complete access to build your [06:07:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22046s) first automations there. Thank you for [06:07:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22047s) your time everyone. [06:07:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=22049s) >> Thank you. [applause]