# What Your AI Doesn't Know Source: https://www.youtube.com/watch?v=y9JtCCZP2MQ Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session. [02:59:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10748s) All right, hello everyone. [02:59:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10749s) My name is Jeff Pinner. [02:59:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10751s) I'm the CTO of EXA. [02:59:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10752s) Thank you guys for having us here. [02:59:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10753s) And we're going to talk a little bit today about things your agent doesn't know. [02:59:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10762s) So Web Search has been around now for about 30 years, long enough where at this point [02:59:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10767s) we're kind of used to it and we don't really question the status quo or how it could be [02:59:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10771s) different with Agents. [02:59:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10774s) Over the last 18 months or so, Search has picked up Agents as sort of a new customer. [02:59:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10779s) They're searching on our behalf. [02:59:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10781s) And so at Exo, we specifically build Web Search as a knowledge layer for Agents. [02:59:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10787s) We've gone from about 12.5 million requests a month two years ago [02:59:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10792s) to a little over 1.3 billion this past October, [02:59:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10796s) so about 100x over two years. [02:59:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10798s) And it's not unique to Exa. [03:00:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10800s) If you talk, OpenAI just announced that ChatGPT handles about a fifth of Google search volumes, and agents in general make up just around half the volume of web traffic now. [03:00:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10811s) And so I think the natural question is, why do agents need search at all? [03:00:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10815s) So let's start with trying to answer this before we talk about how we do search for agents. [03:00:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10824s) So frontier models can do a lot of tasks from parametric knowledge. [03:00:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10830s) But unfortunately for real-world tasks, that simply isn't enough. [03:00:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10835s) Generally, frontier models, no matter how capable they are, are always out of date. [03:00:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10840s) Model launches take two to five months from training cutoff to launch. [03:00:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10846s) Then there's generally a few months after that before the next frontier model is released. [03:00:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10850s) And so taking that together, you're about four to eight months out of date with whatever the latest knowledge base is. [03:00:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10856s) And in the meantime, the world doesn't stop changing just because a model has moved from training to inference. [03:01:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10865s) Libraries launch, APIs change, prices change, new regulations appear, court rulings happen. [03:01:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10872s) The world continues to go. [03:01:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10876s) And even if that weren't true, training data itself is actually just a small amount of [03:01:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10882s) knowledge that the model is trained on. [03:01:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10885s) The biggest models out there are trained on around 50 trillion tokens. [03:01:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10890s) The web itself is around 3,100 trillion. [03:01:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10893s) And that includes the deep web behind things like login and apps. [03:01:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10897s) If you focus it down to just the web that is indexed, the thing that search engines [03:01:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10900s) can see, that's still about 510 trillion tokens. [03:01:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10904s) which means even if the world didn't continue to evolve [03:01:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10908s) after the model cutoff happened, [03:01:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10910s) you would still only be indexing a very small amount [03:01:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10914s) of the knowledge that's out there that models have to go and find. [03:02:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10920s) And for the type of tasks we want models to do, [03:02:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10924s) we want agents to do, [03:02:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10925s) there is valuable work that happens [03:02:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10928s) that is based on outside knowledge, [03:02:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10930s) whether it's across coding, legal work, due diligence work. [03:02:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10935s) Generally, all these tasks require correct, complete knowledge across multiple sources. [03:02:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10941s) And so this is what we're trying to, the gap we're trying to close. [03:02:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10947s) Now, the first thing to do when we're trying to close any gap is to try to measure it. [03:02:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10951s) And so the first thing we do is create a benchmark called Atlas, [03:02:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10956s) which tries to test how well harnesses can leverage real-world knowledge. [03:02:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10962s) So we created the Atlas benchmark to try to test agents in these tasks. [03:02:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10970s) First, we require agents to find entities that match a condition. [03:02:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10976s) Then we require agents to enrich the knowledge with as much information as they can find. [03:03:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10983s) And finally, we score them against the golden answer key. [03:03:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10990s) and we only count rows as correct that correctly identify all aspects of what we're looking for. [03:03:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10996s) And so to succeed at Atlas, you need to be consistently right [03:03:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=10999s) and not just have a decent hit rate for individual searches. [03:03:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11006s) So we're going to use this as a benchmark for how we measure [03:03:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11010s) agent performance when it comes to knowledge retrieval. [03:03:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11013s) And the first thing that we find is that model memory falls off with recency. [03:03:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11020s) When we test agents on Atlas, we see that model success rates fall off sharply as soon as they enter, as soon as they encounter data that's not covered in the training set. [03:03:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11031s) The newer the fact is, the less the models know, and models require, or I should say agents require data to try to achieve these tasks. [03:04:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11041s) And so the first thing that we can see is that memory alone, parametric memory isn't sufficient. [03:04:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11047s) So when we run agents with just parametric memory [03:04:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11050s) without web search, they score around 9% on Atlas, [03:04:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11053s) and so we'll use this to try to understand quantitatively [03:04:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11056s) how web search improves performance. [03:04:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11062s) So the next natural question is, [03:04:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11063s) why don't they just Google it? [03:04:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11068s) And the first reason is agents don't search like people. [03:04:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11072s) Generally, people type about three words into Google, [03:04:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11075s) and that's up from about two words six years ago. [03:04:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11079s) Agents, on the other hand, [03:04:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11081s) search with more or less full sentences. [03:04:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11083s) The mean search on Exa is about 12 words, [03:04:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11086s) about 10% of queries on Exa are 17 words or higher [03:04:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11089s) and these words add constraints, [03:04:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11091s) versioning information, dates, places, [03:04:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11094s) all different types of conditions on the search [03:04:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11096s) that keyword ranking wasn't really designed to handle. [03:04:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11099s) And so we'll take a look at an example of that. [03:05:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11103s) So this is an actual query from EXA. [03:05:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11105s) It's a request for companies. [03:05:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11106s) It's a full sentence. [03:05:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11107s) It adds a bunch of constraints. [03:05:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11109s) And when you feed this into a keyword retrieval algorithm, [03:05:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11113s) what you really get is a handful of results [03:05:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11115s) that don't really answer what the query asked for. [03:05:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11118s) The search didn't understand the query. [03:05:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11121s) The result, the third one, is specifically in the U.S., [03:05:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11124s) and the result asked for data outside the U.S. [03:05:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11129s) And so we're going to come back to this in a few minutes [03:05:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11133s) Because the second thing I want to focus on [03:05:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11135s) Is the fact that for a lot of knowledge on the web [03:05:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11137s) Search just doesn't have access to it [03:05:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11139s) Most of the valuable information isn't on the open web [03:05:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11142s) Reported news, people data, financial data, research [03:05:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11145s) They're all paywalled off [03:05:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11146s) And so search engines that crawl the open internet [03:05:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11150s) Don't ever see these pages [03:05:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11154s) And then lastly, even when knowledge is on the open web [03:05:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11157s) and your agents can access it, [03:05:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11159s) there's a third problem, which is efficiency. [03:06:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11162s) You know, if an agent makes 10 searches, [03:06:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11166s) reads five pages from that search, [03:06:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11168s) gets about 16K of raw HTML from each page, [03:06:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11172s) that's about 860,000 tokens just in processing results. [03:06:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11176s) And if you remember from the first slide, [03:06:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11178s) we serve about 1.3 billion requests a month. [03:06:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11181s) So that's a lot of tokens that are just churning to extract information. [03:06:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11188s) And then finally, when you add keyword search, [03:06:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11191s) and we look at what happens to our Atlas benchmark, [03:06:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11193s) the score definitely improves. [03:06:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11195s) It's added recent knowledge. [03:06:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11197s) But we've gone up to about 28% [03:06:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11199s) because most of the rows still come back with incomplete data [03:06:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11202s) or just wrong values. [03:06:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11205s) And so the challenge when we look at search [03:06:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11207s) is how do we actually provide the knowledge to agents [03:06:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11210s) that they actually require. [03:06:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11216s) So the first thing, [03:06:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11217s) I told you we'd come back to this query, [03:06:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11219s) is, at least at EXA, [03:07:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11221s) the first thing we do to try to tackle this task [03:07:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11224s) is to extract meaning out of information [03:07:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11226s) as opposed to just syntactical matches [03:07:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11228s) or keyword matches. [03:07:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11229s) And so we look through pages, [03:07:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11231s) we understand the intent of queries, [03:07:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11234s) and here we give you results [03:07:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11236s) that specifically answer the intent [03:07:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11238s) with all of the conditionals attached. [03:07:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11242s) And so in order to do this, we do what you would for a modern web crawler. [03:07:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11248s) We crawl the internet, we do a semantic extraction, we embed it in the vector space, we index it. [03:07:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11254s) When your query comes in, we embed the query in the same vector space, and we find the nearest matches. [03:07:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11259s) And this lets us move from syntactical understanding and keyword matching to a semantic understanding of the query in the web. [03:07:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11269s) and we do this at a volume of up to a million pages a second [03:07:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11274s) with a latency of about three minutes from crawl to index. [03:08:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11280s) Now for the second problem, [03:08:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11282s) we need to extend it just beyond the open web. [03:08:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11287s) And so here we have to extend what's on the public web [03:08:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11290s) with both proprietary data that's partnered [03:08:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11297s) and knowledge graph data that we extract and represent. [03:08:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11301s) And this way, the agent can retrieve questions [03:08:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11303s) and answers that it needs to its questions [03:08:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11305s) without having to know where to look for them, [03:08:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11307s) doing this behind a single API. [03:08:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11313s) And for the last challenge, [03:08:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11319s) in order to support the volume of queries that agents make, [03:08:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11322s) We can't simply point them to webpages. [03:08:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11325s) We have to extract the answers from the page [03:08:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11327s) and create a token-dense representation [03:08:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11329s) for agents to process. [03:08:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11331s) It's not sufficient to say the answer to this query [03:08:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11333s) lives on this web page. [03:08:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11336s) You could take, for example, this question [03:08:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11338s) of who won the Nobel Prize for lithium-ion batteries. [03:09:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11340s) If you just point to the Wikipedia page, [03:09:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11343s) you'll get thousands of characters. [03:09:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11345s) The actual excerpt that matters [03:09:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11347s) for citation and grounding is about 500. [03:09:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11349s) So there's about 16x more tokens when you have to process data directly from Google [03:09:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11355s) than when you can extract that data. [03:09:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11357s) And those extractions have to be semantically rich. [03:09:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11360s) You have to pick a segment of the text that answers the question, and so the query matters. [03:09:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11367s) And so what does this actually get us? [03:09:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11370s) When we measure with Atlas, again, to ground ourselves, [03:09:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11374s) we've seen that we've improved from about 9% [03:09:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11376s) on our closed book test [03:09:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11378s) to about 56% when we do a semantic extraction [03:09:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11381s) over the internet and extended knowledge. [03:09:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11385s) So not only do we get about 5x higher of a score, [03:09:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11389s) excuse me, [03:09:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11390s) but we also reduce the cost by about 28% [03:09:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11393s) due to the token efficiency of the extraction. [03:09:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11395s) And so we start to demonstrate quantitatively [03:10:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11400s) that improving access to knowledge actually improves performance of the underlying agent. [03:10:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11406s) And so, yeah, in general, I think the main takeaway that we've hopefully expressed is that while reasoning capabilities on our frontier models are relatively fixed, [03:10:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11420s) A lot of improvement can be done in agentic performance [03:10:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11423s) by extending the harness and giving it access [03:10:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11426s) to semantic meaning across the Internet. [03:10:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11428s) That's it. Thank you. [03:10:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=11431s) Thank