# Beyond Generation: Fast, Structured AI Decisions with Jev at Glean Source: https://www.youtube.com/watch?v=ZZ19zhYnDKY Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session. [05:34:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20073s) Hi everyone. Uh my name is Chennme and I [05:34:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20076s) am a software engineer at clean. I work [05:34:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20080s) on the product engineering side of Glean [05:34:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20082s) assistant. [05:34:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20084s) Today I'm going to give a talk on how [05:34:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20086s) lean uses Jeff. The title of the talk is [05:34:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20090s) beyond generation fast and structured AI [05:34:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20092s) decisions with jean. [05:34:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20096s) If you look at it, most AI product steps [05:34:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20099s) are bounded decisions and not generation [05:35:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20102s) problems. So to give a couple of [05:35:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20104s) examples, [05:35:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20105s) user query intent classification, a [05:35:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20108s) ranker picking a suggestion out of n [05:35:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20110s) options, checking whether a citation is [05:35:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20112s) valid or not, routing to the correct [05:35:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20115s) model or expert, so on so forth. Till [05:35:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20118s) now we have been paying the full L&M [05:35:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20121s) cost and latency to get an answer to all [05:35:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20124s) these bounded problems. [05:35:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20128s) Jeff at Glean fills the gap between [05:35:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20131s) general LLMs and fine-tuned classifiers. [05:35:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20134s) Classification itself isn't a new [05:35:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20136s) problem like people have been trying to [05:35:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20138s) solve it uh since forever. We can have [05:35:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20141s) multiple ways to solve it. Some of the [05:35:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20142s) things that we have tried out include [05:35:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20145s) using a general purpose LLM, [05:35:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20147s) using an open model, something like LIA, [05:35:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20150s) fine-tuning that open model or using [05:35:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20153s) Jeff. [05:35:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20155s) Now all these approaches and all these [05:35:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20157s) uh different ways have their pros and [05:35:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20159s) cons. With a general purpose LLM [05:36:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20162s) we incur a lot of cost and latency while [05:36:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20165s) with something like Jev we get no [05:36:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20166s) generated text output. [05:36:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20169s) Another advantage that we've been seeing [05:36:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20171s) with Jev is that as the taxonomy of the [05:36:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20174s) decision space changes, we don't have to [05:36:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20176s) fine-tune these models again and again. [05:36:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20178s) We can just change our prompt a bit and [05:36:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20180s) boom, it works out of the box. [05:36:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20186s) Now I'll walk you through how we have [05:36:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20189s) integrated gen or these classification [05:36:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20192s) uh pro how we have used Javit glean to [05:36:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20195s) solve these classification problem [05:36:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20196s) statements. I'll go into some of these [05:36:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20198s) problems in great detail. [05:36:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20202s) Before we go into these uh individual [05:36:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20205s) problems, I wanted to touch upon how we [05:36:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20207s) evaluate these changes. Like we have a [05:36:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20209s) huge customer base having millions of [05:36:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20211s) users. So we don't want any of the [05:36:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20213s) change that we make to regress our [05:36:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20214s) product. We have a multi-phase approach, [05:36:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20218s) a rollout approach to any change that we [05:37:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20220s) are making. Uh this includes having a [05:37:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20223s) golden eval set for almost every [05:37:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20225s) possible thing like any place where we [05:37:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20227s) have generative AI, any place where we [05:37:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20229s) have any form of intelligence involved, [05:37:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20231s) we have a golden eval set for that [05:37:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20234s) space. We soak all of these changes for [05:37:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20237s) a significant amount of time for our [05:37:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20238s) internal use case. like we have around [05:37:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20240s) 1500 people at Glean right now. We soak [05:37:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20242s) all of these changes within our Glean [05:37:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20246s) instance for like couple of weeks. We [05:37:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20247s) get active feedback. We have a very [05:37:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20249s) active feedback loop from our internal [05:37:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20251s) users. We work upon that and then we go [05:37:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20253s) for an AB testing [05:37:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20256s) uh with our customers. [05:37:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20259s) The first problem that we tried out with [05:37:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20261s) a decision pro uh decision model was [05:37:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20263s) query classification. [05:37:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20266s) Query classification is when we want to [05:37:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20268s) know the nature of user queries. This [05:37:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20270s) helps us realize what kind of query what [05:37:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20273s) kind of uh jobs are people using clean [05:37:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20276s) for what kind of say for example if [05:37:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20278s) people are using more for sales like [05:38:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20280s) queries we focus more on that. If people [05:38:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20282s) are using more for engineering type use [05:38:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20283s) cases we focus more on that. We tried [05:38:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20286s) out three different approaches over here [05:38:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20287s) and this is uh data from our test bed. [05:38:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20291s) Uh we used an open source layer model. [05:38:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20294s) Then we tried out Jev and then we [05:38:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20296s) fine-tuned the liar model on our own [05:38:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20298s) compute resources. [05:38:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20301s) Jev is the best no training option in [05:38:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20304s) our opinion. It nearly doubles the [05:38:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20306s) output quality of a base layer model and [05:38:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20309s) is 90 is able to perform at 90% quality [05:38:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20312s) in comparison to a fine-tuned model [05:38:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20314s) fine-tuned layer. [05:38:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20316s) With Jeff, we don't have to go through [05:38:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20318s) the very tedious process of data [05:38:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20321s) collection, data cleaning, incurring the [05:38:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20323s) cost for compute, like training the [05:38:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20325s) model, fine-tuning the model again and [05:38:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20328s) again as the taxonomy changes. And at [05:38:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20330s) the same time, we don't have to worry [05:38:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20332s) about hosting the model or uh incuring [05:38:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20334s) the inference cost. [05:38:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20339s) The next problem we went to is expert [05:39:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20341s) routing. This is a pretty problem common [05:39:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20342s) problem statement in the industry right [05:39:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20344s) now like hey when the user query comes [05:39:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20346s) in what expert do we route it to like [05:39:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20348s) how do we ensure that the user uh user [05:39:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20351s) the response to the user is of the best [05:39:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20353s) possible quality glean receives the way [05:39:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20355s) we have been solving this is glean [05:39:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20357s) receives the user query and then decides [05:39:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20359s) what's the best expert to answer the [05:39:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20361s) query till now we have been using a [05:39:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20363s) general purpose LLM to route to the [05:39:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20364s) correct expert we set up a test bed uh [05:39:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20367s) for Jeff we set up a test bed with n [05:39:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20369s) experts and Our experiments found a [05:39:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20372s) significantly high precision when using [05:39:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20373s) jev and that to at an almost 8x [05:39:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20376s) reduction in latency. [05:39:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20379s) We had to change our implementation or [05:39:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20381s) our infrastructure significantly to [05:39:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20383s) integrate je into this specific stage of [05:39:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20386s) our product. But the reduction in [05:39:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20389s) latency justifies that inherent [05:39:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20392s) complexity in the system. We at glean [05:39:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20394s) have been trying to build systems in the [05:39:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20396s) most in the simplest possible manners. [05:39:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20398s) This is one of the rare pieces where we [05:40:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20400s) believe complexity is worth it. [05:40:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20404s) Next, I'll walk you through the ranker [05:40:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20407s) problem statement that I talked about. [05:40:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20410s) When we receive a user query, we also [05:40:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20413s) want to generate the best follow-up [05:40:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20414s) suggestions for the user. Like this [05:40:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20416s) helps us in getting discoverability for [05:40:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20418s) our product. This helps us in ensuring [05:40:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20419s) that the user has the best possible [05:40:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20420s) experience. We also want to recommend or [05:40:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20424s) say preempt what the next user query [05:40:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20426s) might be. Again a pretty pro common [05:40:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20428s) problem statement that the entire [05:40:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20429s) industry has been trying to solve for. [05:40:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20431s) Internally we call this system Q like Q [05:40:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20436s) as in the literal Q. The way this works [05:40:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20439s) we have like 10 or 15 different sources [05:40:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20442s) or 10 or 15 different ways uh 10 or 15 [05:40:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20446s) different sources that are generating [05:40:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20447s) suggestions independently of each other. [05:40:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20449s) Once those suggest these suggestions are [05:40:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20451s) generated by natural language LLMs and [05:40:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20454s) it's like a proper natural language [05:40:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20456s) query. Once these suggestions are in we [05:40:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20459s) have a ranking step. So the ranker takes [05:41:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20461s) in all these suggestions takes in the [05:41:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20463s) you uh the first user query takes in the [05:41:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20465s) output uh that glean gave and then all [05:41:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20468s) these suggestions in that context. The [05:41:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20470s) ranker is then asked to decide like hey [05:41:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20472s) you have these suggestions what do you [05:41:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20473s) think is the best one. A pretty [05:41:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20476s) interesting bottleneck that we had over [05:41:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20478s) here is we want the response to be [05:41:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20479s) snappy. I cannot we cannot have a state [05:41:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20482s) where we give the textual response to [05:41:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20485s) the user and then wait 5 seconds for [05:41:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20487s) generating the follow-up suggestion. We [05:41:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20489s) have to be snappy. That's why we have [05:41:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20492s) time limits on these suggestions. So [05:41:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20494s) right now this step had a time limit of [05:41:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20495s) 1 1.5 to 2 seconds. [05:41:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20498s) Jeff no doubt gave us a better quality [05:41:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20500s) than better quality and far better [05:41:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20501s) latency than what we had with a general [05:41:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20504s) purpose LLM. But two things stood out [05:41:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20506s) over there. First, we were able to give [05:41:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20510s) a suggestion within our time bounds in [05:41:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20513s) 99.2% [05:41:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20515s) cases. With the general purpose LM, this [05:41:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20517s) was almost 90%. So this ensures that we [05:42:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20522s) have user engagement and far more user [05:42:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20525s) queries. Now the second thing is at the [05:42:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20529s) glean whenever we use these LLMs we have [05:42:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20531s) guardrails. Guardrails in the sense that [05:42:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20534s) hey LLM might not be good at generating [05:42:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20536s) a response or LLM might be making some [05:42:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20538s) specific kind of mistakes. So we have [05:42:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20540s) checks and balances for all those [05:42:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20542s) possible mistakes that we see. We keep a [05:42:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20544s) log of like we keep a log of all these [05:42:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20547s) checks that are incurred at any point of [05:42:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20549s) time. With a general purpose LLM, we [05:42:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20553s) were able to generate a valid suggestion [05:42:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20555s) in almost 90% of the use uh 90% of the [05:42:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20558s) cases. We still had some places where [05:42:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20560s) LLM was either generating a langu [05:42:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20561s) generating a response in a different [05:42:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20563s) language uh the output [05:42:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20566s) did not uh meet certain other criterias [05:42:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20568s) that we had uh that we had written [05:42:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20571s) programmatically. With Jev, we found [05:42:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20574s) zero cases to be there when the output [05:42:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20576s) was invalid. Now a natural question is [05:42:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20579s) like hey but you would have used a [05:43:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20580s) different prompt. Yes the prompt was [05:43:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20582s) somewhat different but we tried to mimic [05:43:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20584s) the prompt in the two cases to the [05:43:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20585s) largest extent possible so that we don't [05:43:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20587s) have a divergence. Like again we at [05:43:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20588s) glean want to keep the system as simple [05:43:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20590s) as possible. [05:43:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20594s) Another benefit we see with using a [05:43:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20597s) decision model is how we can control the [05:43:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20599s) thresholds because in this ranker [05:43:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20602s) problem statement we are getting [05:43:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20604s) specific probabilities of each output. [05:43:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20606s) We are not restricted to a binary yes or [05:43:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20608s) no response. We get like hey these were [05:43:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20610s) the five options and these are the [05:43:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20612s) probabilities of these five options. [05:43:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20614s) With this we are able to fine-tune the [05:43:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20616s) thresholds based on the deployment. We [05:43:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20619s) have put in self-arning systems which [05:43:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20621s) based on the user based on the customer [05:43:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20624s) deployment and the uh nature of usage [05:43:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20627s) that the specific customer has [05:43:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20629s) fine-tunes the thresholds by itself. [05:43:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20631s) This ensures that the experience in each [05:43:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20635s) and every customer out there is the best [05:43:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20637s) possible based on how they use glean not [05:44:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20640s) a general purpose solution that fits [05:44:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20642s) all. [05:44:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20648s) Okay, this is an interesting one. Now [05:44:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20650s) moving on from how we have been making [05:44:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20653s) the lives or the experience of our end [05:44:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20655s) users better, we also tried using Jev [05:44:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20658s) and some of the some some other parts of [05:44:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20661s) our uh development pipeline. One of them [05:44:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20663s) was [05:44:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20665s) how we use LLM as judges like all of us [05:44:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20669s) have been using LLM as judges for quite [05:44:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20670s) some time now. There's an input, there's [05:44:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20672s) an output and we use an LLM to judge how [05:44:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20675s) the quality of the output is. Now [05:44:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20677s) naturally whenever we use LLM as a judge [05:44:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20679s) we repeat the experiment multiple times [05:44:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20681s) because to account for the variability [05:44:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20683s) in the response. [05:44:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20685s) At Glean we used Jeff for a very [05:44:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20687s) specific part of our evaluation [05:44:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20688s) pipeline. [05:44:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20690s) Glean responses or like any other AI [05:44:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20692s) product out there has like citations. [05:44:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20694s) This is how people trust that Glean [05:44:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20697s) Glean's answer was correct and it it is [05:44:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20699s) grounded in the correct document. Now [05:45:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20701s) these site we hold these citations to a [05:45:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20704s) very high bar and we have a specific [05:45:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20707s) part in our evaluation pipeline where we [05:45:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20710s) check whether the cited document really [05:45:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20712s) supports the claim made in the answer or [05:45:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20714s) not. This again seems to be a pretty [05:45:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20717s) simple yes or no question like hey [05:45:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20719s) whether the claim is uh supported or [05:45:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20722s) not. With Jev no doubt we had a far [05:45:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20724s) better latency and cost like visible in [05:45:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20727s) the slide uh over here. The interesting [05:45:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20731s) thing we noticed over here is how [05:45:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20733s) consistent and reproducible Jeff results [05:45:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20736s) were in comparison to something like say [05:45:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20737s) Luna in the 28 or 30 use cases that we [05:45:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20741s) had like only one run had like a [05:45:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20744s) difference in response. [05:45:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20746s) Apart from that everything was [05:45:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20748s) consistent but this ensure this this [05:45:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20750s) ensures two things for us as engineers [05:45:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20752s) or us as developers at glean. We are not [05:45:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20755s) only improving the quality of our [05:45:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20757s) product or we are not only improving [05:45:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20759s) glean in a linear fashion anymore. Any [05:46:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20762s) improvement of this extent in the [05:46:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20764s) development pipeline leads to an [05:46:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20766s) exponential increase in how we can [05:46:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20769s) improve the product. So if I'm if I as a [05:46:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20771s) developer I'm able to ship things like [05:46:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20773s) 5x faster like this is a significant [05:46:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20775s) reduction in latency. like if I run this [05:46:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20778s) experiment across thousands and [05:46:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20779s) thousands of entries, this is a lot of [05:46:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20781s) time and again I don't have to run the [05:46:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20783s) same uh input output pair multiple times [05:46:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20786s) because the result from J are quite [05:46:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20787s) consistent. So if I'm able to increase [05:46:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20791s) my own productivity by like 5x or 6x I [05:46:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20794s) can ship 6x times the features to our [05:46:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20796s) customers and this happens across like [05:46:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20798s) 500 engineers or 300 engineers at clean. [05:46:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20801s) So the impact of a a good decision model [05:46:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20805s) is no longer linear. it is exponential [05:46:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20808s) on how we can improve the product. [05:46:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20812s) Okay. [05:46:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20814s) Now, moving on from product and uh LLM [05:46:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20817s) as judges, there's a pretty important [05:47:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20820s) this this is a pretty important part of [05:47:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20823s) how we use our customer data. Like [05:47:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20826s) security is something that all our [05:47:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20828s) customers expect from us. So, for [05:47:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20830s) example, we have checks and balances on [05:47:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20833s) all the inputs and outputs coming from a [05:47:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20835s) tool. all the inputs and outputs uh from [05:47:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20837s) an LLM. [05:47:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20839s) Till now we at glean have like a [05:47:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20841s) multi-phase approach. There is like [05:47:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20844s) there are two phases to this. The first [05:47:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20846s) one is like recall heavy phase and the [05:47:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20848s) second one is the final heavy loaded [05:47:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20850s) L&M. All these calls are made in [05:47:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20852s) parallel to like the actual tool input [05:47:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20854s) output calls. [05:47:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20857s) Till now we used to host a virtue [05:47:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20859s) classifier using T4 GPUs across all our [05:47:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20862s) customers. So glean is a single tenant [05:47:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20864s) solution like we don't use or share [05:47:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20866s) resources across customers. Every [05:47:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20868s) customer gets gets its own deployment [05:47:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20870s) gets its own instance and gets all the [05:47:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20872s) resources. So assuming we have thousand [05:47:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20876s) customers out there we had to maintain [05:47:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20878s) like virtue classifiers across all these [05:48:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20880s) thousand customers which becomes a [05:48:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20882s) pretty difficult problem statement in [05:48:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20883s) itself which takes up a lot of time from [05:48:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20885s) glean engineers. [05:48:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20887s) So looking at the results over here the [05:48:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20890s) overall recall of the first phase where [05:48:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20892s) we used to have a virtual classifier we [05:48:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20894s) replace that with jev went from 31.5% to [05:48:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20897s) 90.2%. [05:48:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20899s) At the same time we are seeing a [05:48:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20900s) significant reduction in cost. Now this [05:48:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20902s) cost is only the resource cost mentioned [05:48:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20905s) over here but we are also saving cost on [05:48:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20908s) the engineer and resources over here. We [05:48:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20910s) don't have to invest like two or three [05:48:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20912s) engineers who are full-time maintaining [05:48:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20914s) those virtual classifiers or those T4 [05:48:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20915s) GPUs across customers. The the [05:48:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20919s) improvement or the savings that we have [05:48:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20921s) are far more than 50%. [05:48:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20926s) This slide concludes all the results I [05:48:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20928s) have talked about till now. Jev at Glean [05:48:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20931s) has been able to influence the product [05:48:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20933s) experience of the users has been able to [05:48:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20936s) uh impact how we as developers have been [05:48:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20938s) working at Glean and most importantly it [05:49:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20941s) also has a very good positive impact on [05:49:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20943s) the security that we provide to our [05:49:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20945s) customers. [05:49:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20946s) A simple rule that we have been using is [05:49:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20948s) can the out in conclusion a simple rule [05:49:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20951s) that we have been using at glean is can [05:49:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20953s) the output be listed in advance if no [05:49:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20955s) then it's a free form gener generative [05:49:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20957s) text problem statement go back to [05:49:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20959s) something that can generate free form [05:49:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20960s) text if yes then use jev are labels [05:49:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20964s) stable and volume high if this is the [05:49:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20966s) case where the labels are stable which [05:49:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20968s) is rarely the case anymore uh you can [05:49:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20971s) also fine-tune a model but if the labels [05:49:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20973s) change which in my opinion is quite [05:49:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20975s) frequent these is then again use [05:49:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20977s) something like jev. [05:49:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20980s) Obviously if you need dynamic text we [05:49:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20982s) can like feel free to try out more [05:49:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20985s) hybrid approaches like jev along with uh [05:49:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20988s) free form text lms. [05:49:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20992s) This ends my presentation. So thank you [05:49:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20994s) everyone for listening. Uh I have added [05:49:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20997s) uh a blog like a link to the blog post [05:49:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=20999s) where we have posted the exact same [05:50:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21001s) results in great detail. Uh, feel free [05:50:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21004s) to catch me on the outside if you have [05:50:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=21005s) more questions. Thank you.