Beyond Generation: Fast, Structured AI Decisions with Jev at Glean

5:34:33Hi everyone. Uh my name is Chennme and I

5:34:36am a software engineer at clean. I work

5:34:40on the product engineering side of Glean

5:34:42assistant.

5:34:44Today I'm going to give a talk on how

5:34:46lean uses Jeff. The title of the talk is

5:34:50beyond generation fast and structured AI

5:34:52decisions with jean.

5:34:56If you look at it, most AI product steps

5:34:59are bounded decisions and not generation

5:35:02problems. So to give a couple of

5:35:04examples,

5:35:05user query intent classification, a

5:35:08ranker picking a suggestion out of n

5:35:10options, checking whether a citation is

5:35:12valid or not, routing to the correct

5:35:15model or expert, so on so forth. Till

5:35:18now we have been paying the full L&M

5:35:21cost and latency to get an answer to all

5:35:24these bounded problems.

5:35:28Jeff at Glean fills the gap between

5:35:31general LLMs and fine-tuned classifiers.

5:35:34Classification itself isn't a new

5:35:36problem like people have been trying to

5:35:38solve it uh since forever. We can have

5:35:41multiple ways to solve it. Some of the

5:35:42things that we have tried out include

5:35:45using a general purpose LLM,

5:35:47using an open model, something like LIA,

5:35:50fine-tuning that open model or using

5:35:53Jeff.

5:35:55Now all these approaches and all these

5:35:57uh different ways have their pros and

5:35:59cons. With a general purpose LLM

5:36:02we incur a lot of cost and latency while

5:36:05with something like Jev we get no

5:36:06generated text output.

5:36:09Another advantage that we've been seeing

5:36:11with Jev is that as the taxonomy of the

5:36:14decision space changes, we don't have to

5:36:16fine-tune these models again and again.

5:36:18We can just change our prompt a bit and

5:36:20boom, it works out of the box.

5:36:26Now I'll walk you through how we have

5:36:29integrated gen or these classification

5:36:32uh pro how we have used Javit glean to

5:36:35solve these classification problem

5:36:36statements. I'll go into some of these

5:36:38problems in great detail.

5:36:42Before we go into these uh individual

5:36:45problems, I wanted to touch upon how we

5:36:47evaluate these changes. Like we have a

5:36:49huge customer base having millions of

5:36:51users. So we don't want any of the

5:36:53change that we make to regress our

5:36:54product. We have a multi-phase approach,

5:36:58a rollout approach to any change that we

5:37:00are making. Uh this includes having a

5:37:03golden eval set for almost every

5:37:05possible thing like any place where we

5:37:07have generative AI, any place where we

5:37:09have any form of intelligence involved,

5:37:11we have a golden eval set for that

5:37:14space. We soak all of these changes for

5:37:17a significant amount of time for our

5:37:18internal use case. like we have around

5:37:201500 people at Glean right now. We soak

5:37:22all of these changes within our Glean

5:37:26instance for like couple of weeks. We

5:37:27get active feedback. We have a very

5:37:29active feedback loop from our internal

5:37:31users. We work upon that and then we go

5:37:33for an AB testing

5:37:36uh with our customers.

5:37:39The first problem that we tried out with

5:37:41a decision pro uh decision model was

5:37:43query classification.

5:37:46Query classification is when we want to

5:37:48know the nature of user queries. This

5:37:50helps us realize what kind of query what

5:37:53kind of uh jobs are people using clean

5:37:56for what kind of say for example if

5:37:58people are using more for sales like

5:38:00queries we focus more on that. If people

5:38:02are using more for engineering type use

5:38:03cases we focus more on that. We tried

5:38:06out three different approaches over here

5:38:07and this is uh data from our test bed.

5:38:11Uh we used an open source layer model.

5:38:14Then we tried out Jev and then we

5:38:16fine-tuned the liar model on our own

5:38:18compute resources.

5:38:21Jev is the best no training option in

5:38:24our opinion. It nearly doubles the

5:38:26output quality of a base layer model and

5:38:29is 90 is able to perform at 90% quality

5:38:32in comparison to a fine-tuned model

5:38:34fine-tuned layer.

5:38:36With Jeff, we don't have to go through

5:38:38the very tedious process of data

5:38:41collection, data cleaning, incurring the

5:38:43cost for compute, like training the

5:38:45model, fine-tuning the model again and

5:38:48again as the taxonomy changes. And at

5:38:50the same time, we don't have to worry

5:38:52about hosting the model or uh incuring

5:38:54the inference cost.

5:38:59The next problem we went to is expert

5:39:01routing. This is a pretty problem common

5:39:02problem statement in the industry right

5:39:04now like hey when the user query comes

5:39:06in what expert do we route it to like

5:39:08how do we ensure that the user uh user

5:39:11the response to the user is of the best

5:39:13possible quality glean receives the way

5:39:15we have been solving this is glean

5:39:17receives the user query and then decides

5:39:19what's the best expert to answer the

5:39:21query till now we have been using a

5:39:23general purpose LLM to route to the

5:39:24correct expert we set up a test bed uh

5:39:27for Jeff we set up a test bed with n

5:39:29experts and Our experiments found a

5:39:32significantly high precision when using

5:39:33jev and that to at an almost 8x

5:39:36reduction in latency.

5:39:39We had to change our implementation or

5:39:41our infrastructure significantly to

5:39:43integrate je into this specific stage of

5:39:46our product. But the reduction in

5:39:49latency justifies that inherent

5:39:52complexity in the system. We at glean

5:39:54have been trying to build systems in the

5:39:56most in the simplest possible manners.

5:39:58This is one of the rare pieces where we

5:40:00believe complexity is worth it.

5:40:04Next, I'll walk you through the ranker

5:40:07problem statement that I talked about.

5:40:10When we receive a user query, we also

5:40:13want to generate the best follow-up

5:40:14suggestions for the user. Like this

5:40:16helps us in getting discoverability for

5:40:18our product. This helps us in ensuring

5:40:19that the user has the best possible

5:40:20experience. We also want to recommend or

5:40:24say preempt what the next user query

5:40:26might be. Again a pretty pro common

5:40:28problem statement that the entire

5:40:29industry has been trying to solve for.

5:40:31Internally we call this system Q like Q

5:40:36as in the literal Q. The way this works

5:40:39we have like 10 or 15 different sources

5:40:42or 10 or 15 different ways uh 10 or 15

5:40:46different sources that are generating

5:40:47suggestions independently of each other.

5:40:49Once those suggest these suggestions are

5:40:51generated by natural language LLMs and

5:40:54it's like a proper natural language

5:40:56query. Once these suggestions are in we

5:40:59have a ranking step. So the ranker takes

5:41:01in all these suggestions takes in the

5:41:03you uh the first user query takes in the

5:41:05output uh that glean gave and then all

5:41:08these suggestions in that context. The

5:41:10ranker is then asked to decide like hey

5:41:12you have these suggestions what do you

5:41:13think is the best one. A pretty

5:41:16interesting bottleneck that we had over

5:41:18here is we want the response to be

5:41:19snappy. I cannot we cannot have a state

5:41:22where we give the textual response to

5:41:25the user and then wait 5 seconds for

5:41:27generating the follow-up suggestion. We

5:41:29have to be snappy. That's why we have

5:41:32time limits on these suggestions. So

5:41:34right now this step had a time limit of

5:41:351 1.5 to 2 seconds.

5:41:38Jeff no doubt gave us a better quality

5:41:40than better quality and far better

5:41:41latency than what we had with a general

5:41:44purpose LLM. But two things stood out

5:41:46over there. First, we were able to give

5:41:50a suggestion within our time bounds in

5:41:5399.2%

5:41:55cases. With the general purpose LM, this

5:41:57was almost 90%. So this ensures that we

5:42:02have user engagement and far more user

5:42:05queries. Now the second thing is at the

5:42:09glean whenever we use these LLMs we have

5:42:11guardrails. Guardrails in the sense that

5:42:14hey LLM might not be good at generating

5:42:16a response or LLM might be making some

5:42:18specific kind of mistakes. So we have

5:42:20checks and balances for all those

5:42:22possible mistakes that we see. We keep a

5:42:24log of like we keep a log of all these

5:42:27checks that are incurred at any point of

5:42:29time. With a general purpose LLM, we

5:42:33were able to generate a valid suggestion

5:42:35in almost 90% of the use uh 90% of the

5:42:38cases. We still had some places where

5:42:40LLM was either generating a langu

5:42:41generating a response in a different

5:42:43language uh the output

5:42:46did not uh meet certain other criterias

5:42:48that we had uh that we had written

5:42:51programmatically. With Jev, we found

5:42:54zero cases to be there when the output

5:42:56was invalid. Now a natural question is

5:42:59like hey but you would have used a

5:43:00different prompt. Yes the prompt was

5:43:02somewhat different but we tried to mimic

5:43:04the prompt in the two cases to the

5:43:05largest extent possible so that we don't

5:43:07have a divergence. Like again we at

5:43:08glean want to keep the system as simple

5:43:10as possible.

5:43:14Another benefit we see with using a

5:43:17decision model is how we can control the

5:43:19thresholds because in this ranker

5:43:22problem statement we are getting

5:43:24specific probabilities of each output.

5:43:26We are not restricted to a binary yes or

5:43:28no response. We get like hey these were

5:43:30the five options and these are the

5:43:32probabilities of these five options.

5:43:34With this we are able to fine-tune the

5:43:36thresholds based on the deployment. We

5:43:39have put in self-arning systems which

5:43:41based on the user based on the customer

5:43:44deployment and the uh nature of usage

5:43:47that the specific customer has

5:43:49fine-tunes the thresholds by itself.

5:43:51This ensures that the experience in each

5:43:55and every customer out there is the best

5:43:57possible based on how they use glean not

5:44:00a general purpose solution that fits

5:44:02all.

5:44:08Okay, this is an interesting one. Now

5:44:10moving on from how we have been making

5:44:13the lives or the experience of our end

5:44:15users better, we also tried using Jev

5:44:18and some of the some some other parts of

5:44:21our uh development pipeline. One of them

5:44:23was

5:44:25how we use LLM as judges like all of us

5:44:29have been using LLM as judges for quite

5:44:30some time now. There's an input, there's

5:44:32an output and we use an LLM to judge how

5:44:35the quality of the output is. Now

5:44:37naturally whenever we use LLM as a judge

5:44:39we repeat the experiment multiple times

5:44:41because to account for the variability

5:44:43in the response.

5:44:45At Glean we used Jeff for a very

5:44:47specific part of our evaluation

5:44:48pipeline.

5:44:50Glean responses or like any other AI

5:44:52product out there has like citations.

5:44:54This is how people trust that Glean

5:44:57Glean's answer was correct and it it is

5:44:59grounded in the correct document. Now

5:45:01these site we hold these citations to a

5:45:04very high bar and we have a specific

5:45:07part in our evaluation pipeline where we

5:45:10check whether the cited document really

5:45:12supports the claim made in the answer or

5:45:14not. This again seems to be a pretty

5:45:17simple yes or no question like hey

5:45:19whether the claim is uh supported or

5:45:22not. With Jev no doubt we had a far

5:45:24better latency and cost like visible in

5:45:27the slide uh over here. The interesting

5:45:31thing we noticed over here is how

5:45:33consistent and reproducible Jeff results

5:45:36were in comparison to something like say

5:45:37Luna in the 28 or 30 use cases that we

5:45:41had like only one run had like a

5:45:44difference in response.

5:45:46Apart from that everything was

5:45:48consistent but this ensure this this

5:45:50ensures two things for us as engineers

5:45:52or us as developers at glean. We are not

5:45:55only improving the quality of our

5:45:57product or we are not only improving

5:45:59glean in a linear fashion anymore. Any

5:46:02improvement of this extent in the

5:46:04development pipeline leads to an

5:46:06exponential increase in how we can

5:46:09improve the product. So if I'm if I as a

5:46:11developer I'm able to ship things like

5:46:135x faster like this is a significant

5:46:15reduction in latency. like if I run this

5:46:18experiment across thousands and

5:46:19thousands of entries, this is a lot of

5:46:21time and again I don't have to run the

5:46:23same uh input output pair multiple times

5:46:26because the result from J are quite

5:46:27consistent. So if I'm able to increase

5:46:31my own productivity by like 5x or 6x I

5:46:34can ship 6x times the features to our

5:46:36customers and this happens across like

5:46:38500 engineers or 300 engineers at clean.

5:46:41So the impact of a a good decision model

5:46:45is no longer linear. it is exponential

5:46:48on how we can improve the product.

5:46:52Okay.

5:46:54Now, moving on from product and uh LLM

5:46:57as judges, there's a pretty important

5:47:00this this is a pretty important part of

5:47:03how we use our customer data. Like

5:47:06security is something that all our

5:47:08customers expect from us. So, for

5:47:10example, we have checks and balances on

5:47:13all the inputs and outputs coming from a

5:47:15tool. all the inputs and outputs uh from

5:47:17an LLM.

5:47:19Till now we at glean have like a

5:47:21multi-phase approach. There is like

5:47:24there are two phases to this. The first

5:47:26one is like recall heavy phase and the

5:47:28second one is the final heavy loaded

5:47:30L&M. All these calls are made in

5:47:32parallel to like the actual tool input

5:47:34output calls.

5:47:37Till now we used to host a virtue

5:47:39classifier using T4 GPUs across all our

5:47:42customers. So glean is a single tenant

5:47:44solution like we don't use or share

5:47:46resources across customers. Every

5:47:48customer gets gets its own deployment

5:47:50gets its own instance and gets all the

5:47:52resources. So assuming we have thousand

5:47:56customers out there we had to maintain

5:47:58like virtue classifiers across all these

5:48:00thousand customers which becomes a

5:48:02pretty difficult problem statement in

5:48:03itself which takes up a lot of time from

5:48:05glean engineers.

5:48:07So looking at the results over here the

5:48:10overall recall of the first phase where

5:48:12we used to have a virtual classifier we

5:48:14replace that with jev went from 31.5% to

5:48:1790.2%.

5:48:19At the same time we are seeing a

5:48:20significant reduction in cost. Now this

5:48:22cost is only the resource cost mentioned

5:48:25over here but we are also saving cost on

5:48:28the engineer and resources over here. We

5:48:30don't have to invest like two or three

5:48:32engineers who are full-time maintaining

5:48:34those virtual classifiers or those T4

5:48:35GPUs across customers. The the

5:48:39improvement or the savings that we have

5:48:41are far more than 50%.

5:48:46This slide concludes all the results I

5:48:48have talked about till now. Jev at Glean

5:48:51has been able to influence the product

5:48:53experience of the users has been able to

5:48:56uh impact how we as developers have been

5:48:58working at Glean and most importantly it

5:49:01also has a very good positive impact on

5:49:03the security that we provide to our

5:49:05customers.

5:49:06A simple rule that we have been using is

5:49:08can the out in conclusion a simple rule

5:49:11that we have been using at glean is can

5:49:13the output be listed in advance if no

5:49:15then it's a free form gener generative

5:49:17text problem statement go back to

5:49:19something that can generate free form

5:49:20text if yes then use jev are labels

5:49:24stable and volume high if this is the

5:49:26case where the labels are stable which

5:49:28is rarely the case anymore uh you can

5:49:31also fine-tune a model but if the labels

5:49:33change which in my opinion is quite

5:49:35frequent these is then again use

5:49:37something like jev.

5:49:40Obviously if you need dynamic text we

5:49:42can like feel free to try out more

5:49:45hybrid approaches like jev along with uh

5:49:48free form text lms.

5:49:52This ends my presentation. So thank you

5:49:54everyone for listening. Uh I have added

5:49:57uh a blog like a link to the blog post

5:49:59where we have posted the exact same

5:50:01results in great detail. Uh, feel free

5:50:04to catch me on the outside if you have

5:50:05more questions. Thank you.