1 05:34:33,440 --> 05:34:36,868 Hi everyone. Uh my name is Chennme and I 2 05:34:36,878 --> 05:34:39,990 am a software engineer at clean. I work 3 05:34:40,000 --> 05:34:42,150 on the product engineering side of Glean 4 05:34:42,160 --> 05:34:43,990 assistant. 5 05:34:44,000 --> 05:34:46,948 Today I'm going to give a talk on how 6 05:34:46,958 --> 05:34:49,990 lean uses Jeff. The title of the talk is 7 05:34:50,000 --> 05:34:52,948 beyond generation fast and structured AI 8 05:34:52,958 --> 05:34:55,990 decisions with jean. 9 05:34:56,000 --> 05:34:59,830 If you look at it, most AI product steps 10 05:34:59,840 --> 05:35:02,150 are bounded decisions and not generation 11 05:35:02,160 --> 05:35:04,310 problems. So to give a couple of 12 05:35:04,320 --> 05:35:05,910 examples, 13 05:35:05,920 --> 05:35:08,628 user query intent classification, a 14 05:35:08,638 --> 05:35:10,310 ranker picking a suggestion out of n 15 05:35:10,320 --> 05:35:12,788 options, checking whether a citation is 16 05:35:12,798 --> 05:35:15,110 valid or not, routing to the correct 17 05:35:15,120 --> 05:35:18,548 model or expert, so on so forth. Till 18 05:35:18,558 --> 05:35:21,830 now we have been paying the full L&M 19 05:35:21,840 --> 05:35:24,708 cost and latency to get an answer to all 20 05:35:24,718 --> 05:35:28,868 these bounded problems. 21 05:35:28,878 --> 05:35:31,270 Jeff at Glean fills the gap between 22 05:35:31,280 --> 05:35:34,948 general LLMs and fine-tuned classifiers. 23 05:35:34,958 --> 05:35:36,628 Classification itself isn't a new 24 05:35:36,638 --> 05:35:38,150 problem like people have been trying to 25 05:35:38,160 --> 05:35:41,030 solve it uh since forever. We can have 26 05:35:41,040 --> 05:35:42,788 multiple ways to solve it. Some of the 27 05:35:42,798 --> 05:35:45,190 things that we have tried out include 28 05:35:45,200 --> 05:35:47,910 using a general purpose LLM, 29 05:35:47,920 --> 05:35:50,628 using an open model, something like LIA, 30 05:35:50,638 --> 05:35:53,190 fine-tuning that open model or using 31 05:35:53,200 --> 05:35:55,190 Jeff. 32 05:35:55,200 --> 05:35:57,590 Now all these approaches and all these 33 05:35:57,600 --> 05:35:59,350 uh different ways have their pros and 34 05:35:59,360 --> 05:36:02,788 cons. With a general purpose LLM 35 05:36:02,798 --> 05:36:05,110 we incur a lot of cost and latency while 36 05:36:05,120 --> 05:36:06,788 with something like Jev we get no 37 05:36:06,798 --> 05:36:09,510 generated text output. 38 05:36:09,520 --> 05:36:11,110 Another advantage that we've been seeing 39 05:36:11,120 --> 05:36:14,390 with Jev is that as the taxonomy of the 40 05:36:14,400 --> 05:36:16,150 decision space changes, we don't have to 41 05:36:16,160 --> 05:36:18,070 fine-tune these models again and again. 42 05:36:18,080 --> 05:36:20,788 We can just change our prompt a bit and 43 05:36:20,798 --> 05:36:26,708 boom, it works out of the box. 44 05:36:26,718 --> 05:36:29,190 Now I'll walk you through how we have 45 05:36:29,200 --> 05:36:32,230 integrated gen or these classification 46 05:36:32,240 --> 05:36:35,030 uh pro how we have used Javit glean to 47 05:36:35,040 --> 05:36:36,310 solve these classification problem 48 05:36:36,320 --> 05:36:38,788 statements. I'll go into some of these 49 05:36:38,798 --> 05:36:42,868 problems in great detail. 50 05:36:42,878 --> 05:36:45,110 Before we go into these uh individual 51 05:36:45,120 --> 05:36:47,750 problems, I wanted to touch upon how we 52 05:36:47,760 --> 05:36:49,910 evaluate these changes. Like we have a 53 05:36:49,920 --> 05:36:51,510 huge customer base having millions of 54 05:36:51,520 --> 05:36:53,350 users. So we don't want any of the 55 05:36:53,360 --> 05:36:54,868 change that we make to regress our 56 05:36:54,878 --> 05:36:58,070 product. We have a multi-phase approach, 57 05:36:58,080 --> 05:37:00,310 a rollout approach to any change that we 58 05:37:00,320 --> 05:37:03,670 are making. Uh this includes having a 59 05:37:03,680 --> 05:37:05,750 golden eval set for almost every 60 05:37:05,760 --> 05:37:07,670 possible thing like any place where we 61 05:37:07,680 --> 05:37:09,510 have generative AI, any place where we 62 05:37:09,520 --> 05:37:11,910 have any form of intelligence involved, 63 05:37:11,920 --> 05:37:14,310 we have a golden eval set for that 64 05:37:14,320 --> 05:37:17,030 space. We soak all of these changes for 65 05:37:17,040 --> 05:37:18,868 a significant amount of time for our 66 05:37:18,878 --> 05:37:20,628 internal use case. like we have around 67 05:37:20,638 --> 05:37:22,948 1500 people at Glean right now. We soak 68 05:37:22,958 --> 05:37:26,070 all of these changes within our Glean 69 05:37:26,080 --> 05:37:27,830 instance for like couple of weeks. We 70 05:37:27,840 --> 05:37:29,670 get active feedback. We have a very 71 05:37:29,680 --> 05:37:31,270 active feedback loop from our internal 72 05:37:31,280 --> 05:37:33,750 users. We work upon that and then we go 73 05:37:33,760 --> 05:37:36,230 for an AB testing 74 05:37:36,240 --> 05:37:39,750 uh with our customers. 75 05:37:39,760 --> 05:37:41,430 The first problem that we tried out with 76 05:37:41,440 --> 05:37:43,590 a decision pro uh decision model was 77 05:37:43,600 --> 05:37:46,310 query classification. 78 05:37:46,320 --> 05:37:48,150 Query classification is when we want to 79 05:37:48,160 --> 05:37:49,990 know the nature of user queries. This 80 05:37:50,000 --> 05:37:53,910 helps us realize what kind of query what 81 05:37:53,920 --> 05:37:56,708 kind of uh jobs are people using clean 82 05:37:56,718 --> 05:37:58,548 for what kind of say for example if 83 05:37:58,558 --> 05:37:59,990 people are using more for sales like 84 05:38:00,000 --> 05:38:01,990 queries we focus more on that. If people 85 05:38:02,000 --> 05:38:03,910 are using more for engineering type use 86 05:38:03,920 --> 05:38:06,390 cases we focus more on that. We tried 87 05:38:06,400 --> 05:38:07,750 out three different approaches over here 88 05:38:07,760 --> 05:38:11,830 and this is uh data from our test bed. 89 05:38:11,840 --> 05:38:14,628 Uh we used an open source layer model. 90 05:38:14,638 --> 05:38:16,310 Then we tried out Jev and then we 91 05:38:16,320 --> 05:38:18,868 fine-tuned the liar model on our own 92 05:38:18,878 --> 05:38:21,910 compute resources. 93 05:38:21,920 --> 05:38:24,310 Jev is the best no training option in 94 05:38:24,320 --> 05:38:26,708 our opinion. It nearly doubles the 95 05:38:26,718 --> 05:38:29,030 output quality of a base layer model and 96 05:38:29,040 --> 05:38:32,470 is 90 is able to perform at 90% quality 97 05:38:32,480 --> 05:38:34,390 in comparison to a fine-tuned model 98 05:38:34,400 --> 05:38:36,788 fine-tuned layer. 99 05:38:36,798 --> 05:38:38,390 With Jeff, we don't have to go through 100 05:38:38,400 --> 05:38:41,430 the very tedious process of data 101 05:38:41,440 --> 05:38:43,910 collection, data cleaning, incurring the 102 05:38:43,920 --> 05:38:45,910 cost for compute, like training the 103 05:38:45,920 --> 05:38:48,070 model, fine-tuning the model again and 104 05:38:48,080 --> 05:38:50,868 again as the taxonomy changes. And at 105 05:38:50,878 --> 05:38:52,230 the same time, we don't have to worry 106 05:38:52,240 --> 05:38:54,788 about hosting the model or uh incuring 107 05:38:54,798 --> 05:38:59,030 the inference cost. 108 05:38:59,040 --> 05:39:01,190 The next problem we went to is expert 109 05:39:01,200 --> 05:39:02,868 routing. This is a pretty problem common 110 05:39:02,878 --> 05:39:04,230 problem statement in the industry right 111 05:39:04,240 --> 05:39:06,150 now like hey when the user query comes 112 05:39:06,160 --> 05:39:08,548 in what expert do we route it to like 113 05:39:08,558 --> 05:39:11,670 how do we ensure that the user uh user 114 05:39:11,680 --> 05:39:13,190 the response to the user is of the best 115 05:39:13,200 --> 05:39:15,910 possible quality glean receives the way 116 05:39:15,920 --> 05:39:17,750 we have been solving this is glean 117 05:39:17,760 --> 05:39:19,910 receives the user query and then decides 118 05:39:19,920 --> 05:39:21,270 what's the best expert to answer the 119 05:39:21,280 --> 05:39:23,190 query till now we have been using a 120 05:39:23,200 --> 05:39:24,708 general purpose LLM to route to the 121 05:39:24,718 --> 05:39:27,910 correct expert we set up a test bed uh 122 05:39:27,920 --> 05:39:29,670 for Jeff we set up a test bed with n 123 05:39:29,680 --> 05:39:32,230 experts and Our experiments found a 124 05:39:32,240 --> 05:39:33,910 significantly high precision when using 125 05:39:33,920 --> 05:39:36,310 jev and that to at an almost 8x 126 05:39:36,320 --> 05:39:39,110 reduction in latency. 127 05:39:39,120 --> 05:39:41,910 We had to change our implementation or 128 05:39:41,920 --> 05:39:43,910 our infrastructure significantly to 129 05:39:43,920 --> 05:39:46,070 integrate je into this specific stage of 130 05:39:46,080 --> 05:39:49,510 our product. But the reduction in 131 05:39:49,520 --> 05:39:52,150 latency justifies that inherent 132 05:39:52,160 --> 05:39:54,708 complexity in the system. We at glean 133 05:39:54,718 --> 05:39:56,548 have been trying to build systems in the 134 05:39:56,558 --> 05:39:58,390 most in the simplest possible manners. 135 05:39:58,400 --> 05:40:00,310 This is one of the rare pieces where we 136 05:40:00,320 --> 05:40:04,470 believe complexity is worth it. 137 05:40:04,480 --> 05:40:07,030 Next, I'll walk you through the ranker 138 05:40:07,040 --> 05:40:10,628 problem statement that I talked about. 139 05:40:10,638 --> 05:40:13,190 When we receive a user query, we also 140 05:40:13,200 --> 05:40:14,628 want to generate the best follow-up 141 05:40:14,638 --> 05:40:16,390 suggestions for the user. Like this 142 05:40:16,400 --> 05:40:18,230 helps us in getting discoverability for 143 05:40:18,240 --> 05:40:19,590 our product. This helps us in ensuring 144 05:40:19,600 --> 05:40:20,868 that the user has the best possible 145 05:40:20,878 --> 05:40:24,470 experience. We also want to recommend or 146 05:40:24,480 --> 05:40:26,310 say preempt what the next user query 147 05:40:26,320 --> 05:40:28,548 might be. Again a pretty pro common 148 05:40:28,558 --> 05:40:29,670 problem statement that the entire 149 05:40:29,680 --> 05:40:31,830 industry has been trying to solve for. 150 05:40:31,840 --> 05:40:36,310 Internally we call this system Q like Q 151 05:40:36,320 --> 05:40:39,910 as in the literal Q. The way this works 152 05:40:39,920 --> 05:40:42,708 we have like 10 or 15 different sources 153 05:40:42,718 --> 05:40:45,990 or 10 or 15 different ways uh 10 or 15 154 05:40:46,000 --> 05:40:47,270 different sources that are generating 155 05:40:47,280 --> 05:40:49,830 suggestions independently of each other. 156 05:40:49,840 --> 05:40:51,590 Once those suggest these suggestions are 157 05:40:51,600 --> 05:40:54,628 generated by natural language LLMs and 158 05:40:54,638 --> 05:40:56,788 it's like a proper natural language 159 05:40:56,798 --> 05:40:59,670 query. Once these suggestions are in we 160 05:40:59,680 --> 05:41:01,670 have a ranking step. So the ranker takes 161 05:41:01,680 --> 05:41:03,110 in all these suggestions takes in the 162 05:41:03,120 --> 05:41:05,830 you uh the first user query takes in the 163 05:41:05,840 --> 05:41:08,548 output uh that glean gave and then all 164 05:41:08,558 --> 05:41:10,948 these suggestions in that context. The 165 05:41:10,958 --> 05:41:12,628 ranker is then asked to decide like hey 166 05:41:12,638 --> 05:41:13,910 you have these suggestions what do you 167 05:41:13,920 --> 05:41:16,310 think is the best one. A pretty 168 05:41:16,320 --> 05:41:18,070 interesting bottleneck that we had over 169 05:41:18,080 --> 05:41:19,830 here is we want the response to be 170 05:41:19,840 --> 05:41:22,310 snappy. I cannot we cannot have a state 171 05:41:22,320 --> 05:41:25,030 where we give the textual response to 172 05:41:25,040 --> 05:41:27,830 the user and then wait 5 seconds for 173 05:41:27,840 --> 05:41:29,750 generating the follow-up suggestion. We 174 05:41:29,760 --> 05:41:32,150 have to be snappy. That's why we have 175 05:41:32,160 --> 05:41:34,070 time limits on these suggestions. So 176 05:41:34,080 --> 05:41:35,750 right now this step had a time limit of 177 05:41:35,760 --> 05:41:38,470 1 1.5 to 2 seconds. 178 05:41:38,480 --> 05:41:40,230 Jeff no doubt gave us a better quality 179 05:41:40,240 --> 05:41:41,830 than better quality and far better 180 05:41:41,840 --> 05:41:44,070 latency than what we had with a general 181 05:41:44,080 --> 05:41:46,948 purpose LLM. But two things stood out 182 05:41:46,958 --> 05:41:50,548 over there. First, we were able to give 183 05:41:50,558 --> 05:41:53,590 a suggestion within our time bounds in 184 05:41:53,600 --> 05:41:55,190 99.2% 185 05:41:55,200 --> 05:41:57,910 cases. With the general purpose LM, this 186 05:41:57,920 --> 05:42:02,868 was almost 90%. So this ensures that we 187 05:42:02,878 --> 05:42:05,830 have user engagement and far more user 188 05:42:05,840 --> 05:42:09,510 queries. Now the second thing is at the 189 05:42:09,520 --> 05:42:11,830 glean whenever we use these LLMs we have 190 05:42:11,840 --> 05:42:14,390 guardrails. Guardrails in the sense that 191 05:42:14,400 --> 05:42:16,070 hey LLM might not be good at generating 192 05:42:16,080 --> 05:42:18,070 a response or LLM might be making some 193 05:42:18,080 --> 05:42:20,548 specific kind of mistakes. So we have 194 05:42:20,558 --> 05:42:22,230 checks and balances for all those 195 05:42:22,240 --> 05:42:24,390 possible mistakes that we see. We keep a 196 05:42:24,400 --> 05:42:27,270 log of like we keep a log of all these 197 05:42:27,280 --> 05:42:29,430 checks that are incurred at any point of 198 05:42:29,440 --> 05:42:33,590 time. With a general purpose LLM, we 199 05:42:33,600 --> 05:42:35,750 were able to generate a valid suggestion 200 05:42:35,760 --> 05:42:38,310 in almost 90% of the use uh 90% of the 201 05:42:38,320 --> 05:42:40,230 cases. We still had some places where 202 05:42:40,240 --> 05:42:41,910 LLM was either generating a langu 203 05:42:41,920 --> 05:42:43,110 generating a response in a different 204 05:42:43,120 --> 05:42:46,470 language uh the output 205 05:42:46,480 --> 05:42:48,708 did not uh meet certain other criterias 206 05:42:48,718 --> 05:42:51,190 that we had uh that we had written 207 05:42:51,200 --> 05:42:54,150 programmatically. With Jev, we found 208 05:42:54,160 --> 05:42:56,628 zero cases to be there when the output 209 05:42:56,638 --> 05:42:59,430 was invalid. Now a natural question is 210 05:42:59,440 --> 05:43:00,868 like hey but you would have used a 211 05:43:00,878 --> 05:43:02,310 different prompt. Yes the prompt was 212 05:43:02,320 --> 05:43:04,390 somewhat different but we tried to mimic 213 05:43:04,400 --> 05:43:05,590 the prompt in the two cases to the 214 05:43:05,600 --> 05:43:07,110 largest extent possible so that we don't 215 05:43:07,120 --> 05:43:08,948 have a divergence. Like again we at 216 05:43:08,958 --> 05:43:10,708 glean want to keep the system as simple 217 05:43:10,718 --> 05:43:14,788 as possible. 218 05:43:14,798 --> 05:43:17,030 Another benefit we see with using a 219 05:43:17,040 --> 05:43:19,750 decision model is how we can control the 220 05:43:19,760 --> 05:43:22,390 thresholds because in this ranker 221 05:43:22,400 --> 05:43:23,990 problem statement we are getting 222 05:43:24,000 --> 05:43:26,310 specific probabilities of each output. 223 05:43:26,320 --> 05:43:28,868 We are not restricted to a binary yes or 224 05:43:28,878 --> 05:43:30,708 no response. We get like hey these were 225 05:43:30,718 --> 05:43:32,390 the five options and these are the 226 05:43:32,400 --> 05:43:34,548 probabilities of these five options. 227 05:43:34,558 --> 05:43:36,470 With this we are able to fine-tune the 228 05:43:36,480 --> 05:43:39,190 thresholds based on the deployment. We 229 05:43:39,200 --> 05:43:41,830 have put in self-arning systems which 230 05:43:41,840 --> 05:43:44,070 based on the user based on the customer 231 05:43:44,080 --> 05:43:47,430 deployment and the uh nature of usage 232 05:43:47,440 --> 05:43:49,670 that the specific customer has 233 05:43:49,680 --> 05:43:51,590 fine-tunes the thresholds by itself. 234 05:43:51,600 --> 05:43:55,030 This ensures that the experience in each 235 05:43:55,040 --> 05:43:57,430 and every customer out there is the best 236 05:43:57,440 --> 05:44:00,548 possible based on how they use glean not 237 05:44:00,558 --> 05:44:02,708 a general purpose solution that fits 238 05:44:02,718 --> 05:44:08,070 all. 239 05:44:08,080 --> 05:44:10,788 Okay, this is an interesting one. Now 240 05:44:10,798 --> 05:44:13,430 moving on from how we have been making 241 05:44:13,440 --> 05:44:15,430 the lives or the experience of our end 242 05:44:15,440 --> 05:44:18,788 users better, we also tried using Jev 243 05:44:18,798 --> 05:44:21,190 and some of the some some other parts of 244 05:44:21,200 --> 05:44:23,670 our uh development pipeline. One of them 245 05:44:23,680 --> 05:44:25,270 was 246 05:44:25,280 --> 05:44:29,190 how we use LLM as judges like all of us 247 05:44:29,200 --> 05:44:30,868 have been using LLM as judges for quite 248 05:44:30,878 --> 05:44:32,708 some time now. There's an input, there's 249 05:44:32,718 --> 05:44:35,430 an output and we use an LLM to judge how 250 05:44:35,440 --> 05:44:37,430 the quality of the output is. Now 251 05:44:37,440 --> 05:44:39,750 naturally whenever we use LLM as a judge 252 05:44:39,760 --> 05:44:41,670 we repeat the experiment multiple times 253 05:44:41,680 --> 05:44:43,350 because to account for the variability 254 05:44:43,360 --> 05:44:45,430 in the response. 255 05:44:45,440 --> 05:44:47,510 At Glean we used Jeff for a very 256 05:44:47,520 --> 05:44:48,868 specific part of our evaluation 257 05:44:48,878 --> 05:44:50,390 pipeline. 258 05:44:50,400 --> 05:44:52,788 Glean responses or like any other AI 259 05:44:52,798 --> 05:44:54,708 product out there has like citations. 260 05:44:54,718 --> 05:44:57,110 This is how people trust that Glean 261 05:44:57,120 --> 05:44:59,350 Glean's answer was correct and it it is 262 05:44:59,360 --> 05:45:01,910 grounded in the correct document. Now 263 05:45:01,920 --> 05:45:04,470 these site we hold these citations to a 264 05:45:04,480 --> 05:45:07,030 very high bar and we have a specific 265 05:45:07,040 --> 05:45:09,990 part in our evaluation pipeline where we 266 05:45:10,000 --> 05:45:12,708 check whether the cited document really 267 05:45:12,718 --> 05:45:14,948 supports the claim made in the answer or 268 05:45:14,958 --> 05:45:17,430 not. This again seems to be a pretty 269 05:45:17,440 --> 05:45:19,270 simple yes or no question like hey 270 05:45:19,280 --> 05:45:21,990 whether the claim is uh supported or 271 05:45:22,000 --> 05:45:24,948 not. With Jev no doubt we had a far 272 05:45:24,958 --> 05:45:27,590 better latency and cost like visible in 273 05:45:27,600 --> 05:45:31,510 the slide uh over here. The interesting 274 05:45:31,520 --> 05:45:33,750 thing we noticed over here is how 275 05:45:33,760 --> 05:45:36,070 consistent and reproducible Jeff results 276 05:45:36,080 --> 05:45:37,750 were in comparison to something like say 277 05:45:37,760 --> 05:45:41,430 Luna in the 28 or 30 use cases that we 278 05:45:41,440 --> 05:45:44,390 had like only one run had like a 279 05:45:44,400 --> 05:45:46,708 difference in response. 280 05:45:46,718 --> 05:45:48,230 Apart from that everything was 281 05:45:48,240 --> 05:45:50,390 consistent but this ensure this this 282 05:45:50,400 --> 05:45:52,868 ensures two things for us as engineers 283 05:45:52,878 --> 05:45:55,510 or us as developers at glean. We are not 284 05:45:55,520 --> 05:45:57,750 only improving the quality of our 285 05:45:57,760 --> 05:45:59,350 product or we are not only improving 286 05:45:59,360 --> 05:46:02,230 glean in a linear fashion anymore. Any 287 05:46:02,240 --> 05:46:04,628 improvement of this extent in the 288 05:46:04,638 --> 05:46:06,390 development pipeline leads to an 289 05:46:06,400 --> 05:46:09,030 exponential increase in how we can 290 05:46:09,040 --> 05:46:11,430 improve the product. So if I'm if I as a 291 05:46:11,440 --> 05:46:13,110 developer I'm able to ship things like 292 05:46:13,120 --> 05:46:15,430 5x faster like this is a significant 293 05:46:15,440 --> 05:46:18,628 reduction in latency. like if I run this 294 05:46:18,638 --> 05:46:19,750 experiment across thousands and 295 05:46:19,760 --> 05:46:21,510 thousands of entries, this is a lot of 296 05:46:21,520 --> 05:46:23,670 time and again I don't have to run the 297 05:46:23,680 --> 05:46:26,310 same uh input output pair multiple times 298 05:46:26,320 --> 05:46:27,910 because the result from J are quite 299 05:46:27,920 --> 05:46:31,030 consistent. So if I'm able to increase 300 05:46:31,040 --> 05:46:34,070 my own productivity by like 5x or 6x I 301 05:46:34,080 --> 05:46:36,788 can ship 6x times the features to our 302 05:46:36,798 --> 05:46:38,628 customers and this happens across like 303 05:46:38,638 --> 05:46:41,590 500 engineers or 300 engineers at clean. 304 05:46:41,600 --> 05:46:45,590 So the impact of a a good decision model 305 05:46:45,600 --> 05:46:48,708 is no longer linear. it is exponential 306 05:46:48,718 --> 05:46:52,230 on how we can improve the product. 307 05:46:52,240 --> 05:46:54,070 Okay. 308 05:46:54,080 --> 05:46:57,350 Now, moving on from product and uh LLM 309 05:46:57,360 --> 05:47:00,548 as judges, there's a pretty important 310 05:47:00,558 --> 05:47:03,110 this this is a pretty important part of 311 05:47:03,120 --> 05:47:06,390 how we use our customer data. Like 312 05:47:06,400 --> 05:47:08,150 security is something that all our 313 05:47:08,160 --> 05:47:10,628 customers expect from us. So, for 314 05:47:10,638 --> 05:47:13,750 example, we have checks and balances on 315 05:47:13,760 --> 05:47:15,510 all the inputs and outputs coming from a 316 05:47:15,520 --> 05:47:17,910 tool. all the inputs and outputs uh from 317 05:47:17,920 --> 05:47:19,670 an LLM. 318 05:47:19,680 --> 05:47:21,830 Till now we at glean have like a 319 05:47:21,840 --> 05:47:24,548 multi-phase approach. There is like 320 05:47:24,558 --> 05:47:26,708 there are two phases to this. The first 321 05:47:26,718 --> 05:47:28,230 one is like recall heavy phase and the 322 05:47:28,240 --> 05:47:29,990 second one is the final heavy loaded 323 05:47:30,000 --> 05:47:32,230 L&M. All these calls are made in 324 05:47:32,240 --> 05:47:34,708 parallel to like the actual tool input 325 05:47:34,718 --> 05:47:37,350 output calls. 326 05:47:37,360 --> 05:47:39,670 Till now we used to host a virtue 327 05:47:39,680 --> 05:47:42,150 classifier using T4 GPUs across all our 328 05:47:42,160 --> 05:47:44,150 customers. So glean is a single tenant 329 05:47:44,160 --> 05:47:46,310 solution like we don't use or share 330 05:47:46,320 --> 05:47:47,990 resources across customers. Every 331 05:47:48,000 --> 05:47:50,470 customer gets gets its own deployment 332 05:47:50,480 --> 05:47:52,628 gets its own instance and gets all the 333 05:47:52,638 --> 05:47:56,070 resources. So assuming we have thousand 334 05:47:56,080 --> 05:47:58,310 customers out there we had to maintain 335 05:47:58,320 --> 05:48:00,230 like virtue classifiers across all these 336 05:48:00,240 --> 05:48:02,390 thousand customers which becomes a 337 05:48:02,400 --> 05:48:03,830 pretty difficult problem statement in 338 05:48:03,840 --> 05:48:05,510 itself which takes up a lot of time from 339 05:48:05,520 --> 05:48:07,590 glean engineers. 340 05:48:07,600 --> 05:48:10,230 So looking at the results over here the 341 05:48:10,240 --> 05:48:12,390 overall recall of the first phase where 342 05:48:12,400 --> 05:48:14,390 we used to have a virtual classifier we 343 05:48:14,400 --> 05:48:17,270 replace that with jev went from 31.5% to 344 05:48:17,280 --> 05:48:19,350 90.2%. 345 05:48:19,360 --> 05:48:20,948 At the same time we are seeing a 346 05:48:20,958 --> 05:48:22,948 significant reduction in cost. Now this 347 05:48:22,958 --> 05:48:25,750 cost is only the resource cost mentioned 348 05:48:25,760 --> 05:48:28,470 over here but we are also saving cost on 349 05:48:28,480 --> 05:48:30,788 the engineer and resources over here. We 350 05:48:30,798 --> 05:48:32,390 don't have to invest like two or three 351 05:48:32,400 --> 05:48:33,990 engineers who are full-time maintaining 352 05:48:34,000 --> 05:48:35,750 those virtual classifiers or those T4 353 05:48:35,760 --> 05:48:39,270 GPUs across customers. The the 354 05:48:39,280 --> 05:48:41,750 improvement or the savings that we have 355 05:48:41,760 --> 05:48:46,390 are far more than 50%. 356 05:48:46,400 --> 05:48:48,788 This slide concludes all the results I 357 05:48:48,798 --> 05:48:51,830 have talked about till now. Jev at Glean 358 05:48:51,840 --> 05:48:53,670 has been able to influence the product 359 05:48:53,680 --> 05:48:56,470 experience of the users has been able to 360 05:48:56,480 --> 05:48:58,788 uh impact how we as developers have been 361 05:48:58,798 --> 05:49:01,670 working at Glean and most importantly it 362 05:49:01,680 --> 05:49:03,510 also has a very good positive impact on 363 05:49:03,520 --> 05:49:05,030 the security that we provide to our 364 05:49:05,040 --> 05:49:06,868 customers. 365 05:49:06,878 --> 05:49:08,948 A simple rule that we have been using is 366 05:49:08,958 --> 05:49:11,270 can the out in conclusion a simple rule 367 05:49:11,280 --> 05:49:13,270 that we have been using at glean is can 368 05:49:13,280 --> 05:49:15,590 the output be listed in advance if no 369 05:49:15,600 --> 05:49:17,110 then it's a free form gener generative 370 05:49:17,120 --> 05:49:19,190 text problem statement go back to 371 05:49:19,200 --> 05:49:20,868 something that can generate free form 372 05:49:20,878 --> 05:49:24,070 text if yes then use jev are labels 373 05:49:24,080 --> 05:49:26,708 stable and volume high if this is the 374 05:49:26,718 --> 05:49:28,788 case where the labels are stable which 375 05:49:28,798 --> 05:49:31,910 is rarely the case anymore uh you can 376 05:49:31,920 --> 05:49:33,670 also fine-tune a model but if the labels 377 05:49:33,680 --> 05:49:35,590 change which in my opinion is quite 378 05:49:35,600 --> 05:49:37,830 frequent these is then again use 379 05:49:37,840 --> 05:49:40,070 something like jev. 380 05:49:40,080 --> 05:49:42,948 Obviously if you need dynamic text we 381 05:49:42,958 --> 05:49:45,270 can like feel free to try out more 382 05:49:45,280 --> 05:49:48,070 hybrid approaches like jev along with uh 383 05:49:48,080 --> 05:49:52,230 free form text lms. 384 05:49:52,240 --> 05:49:54,310 This ends my presentation. So thank you 385 05:49:54,320 --> 05:49:57,430 everyone for listening. Uh I have added 386 05:49:57,440 --> 05:49:59,350 uh a blog like a link to the blog post 387 05:49:59,360 --> 05:50:01,110 where we have posted the exact same 388 05:50:01,120 --> 05:50:04,310 results in great detail. Uh, feel free 389 05:50:04,320 --> 05:50:05,830 to catch me on the outside if you have 390 05:50:05,840 --> 05:50:08,000 more questions. Thank you.