AI Engineer World's Fair 2025
360Brew: LLM-based Personalized Ranking and Recommendation — Hamed Firooz and Maziar Sanjabi, LinkedIn AI
Read the talk
360Brew: From member history to a shared recommendation model
LinkedIn’s 360Brew turns profiles and interactions into prompts, then combines large-model training, gradual compression and sparse attention to make personalized ranking practical.
From a talk by Hamed Firooz and Maziar Sanjabi
Before you start: Familiarity with LLM prompts, transformer attention and basic recommendation systems will help; the article explains the relevant compression and serving choices.
One model for the feed, jobs and search
When someone opens a feed to read an article or searches for a job, the system must find content relevant to that person’s interests. Search and online shopping pose similar matching problems, yet their rankers are often trained and optimized separately. Each new architecture then has to make its way through multiple development and deployment processes. Could one model handle those tasks together?
That is the starting point for 360Brew, the personalization project Hamed Firooz and Maziar Sanjabi describe. The aim is a foundation model that understands a member’s journey across LinkedIn well enough to support its different personalization tasks. Three capabilities would make that consolidation useful:
- Zero-shot recommendation: Describe a new task, its user and its candidate entity, then ask about relevance—without first collecting a task-specific dataset and deploying another ranker.
- In-context learning: Give the model a few interactions, or a description of a member’s interests, to support personalization when behavioral history is sparse.
- Instruction following: Let members specify niche interests or topics they want to explore, and use those instructions to guide recommendations.
These are different ways to adapt the same model: change the task description, provide examples, or express a preference directly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn member behavior into a prediction prompt
The first implementation step is promptification: turn profiles, history and recorded interactions into text the LLM can consume. The prompt begins with an instruction defining the task, adds information about the member, supplies past interactions, and finally introduces a new item. The prediction question is what the member will do with that item. This makes the task instruction part of the input rather than fixing the task entirely in a separate model’s architecture.
For a feed-engagement task, the structure can be expressed as a prompt like this:
Task: Predict whether the member will engage with the candidate post.
Member profile:
Software engineer interested in database performance.
Past interactions, oldest to newest:
- Liked a post about query optimization.
- Was shown a post about office furniture; did not engage.
Candidate post:
How database indexes change query execution plans.
Question:
Will this member engage with the candidate post?
The profile, observed responses and candidate all remain explicit. A general-purpose LLM can already do something with that information, Sanjabi explains, but adequate recommendation quality requires training for the problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the teacher before shrinking the model
The development pipeline separates maximizing quality from maximizing serving efficiency:
- Start with an open-source model. Apply upcycling to control model size and the tradeoff between throughput and quality.
- Train for the domain and tasks. Run continued pretraining, fine-tuning, instruction fine-tuning and alignment.
- Establish a large teacher. Sanjabi describes Brew-XL here as a 150-billion-parameter model optimized for quality.
- Distill into a smaller student. He gives a 3B model as an example of a size that can be made practical for serving.
The large model is an intermediate asset: recommendation traffic demands too many predictions to treat the highest-quality teacher as the default online serving model.
The number of stages makes automation part of the model-development method. Repeatedly moving through training, alignment and distillation would otherwise make iteration cumbersome. Sanjabi says the team also tried avoiding the large teacher, but directly training a small model did not work as well. First establish the capability at large scale, then transfer it into a cheaper model is the recipe their experiments supported.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scale data, model capacity and member history
There are three distinct ways to improve the model. The first is training-data scale. Recommendation systems may have six months, a year or more of behavioral logs, depending on what they record. In the experiment Sanjabi presents, increasing the amount of training data improves performance. That makes the available interaction history a resource for continued model development, not merely an input to individual predictions.
The second lever is model capacity. Sanjabi reports improved performance in a Mixtral architecture experiment spanning 7B to 8×22B. This is especially useful for the XL teacher, where the immediate objective is quality before compression.
The third lever is context length, which determines how much of a member’s history can accompany a prediction. Supplying more history initially improves performance in the presented experiment, but the curve eventually turns downward. Sanjabi attributes that decline to the tested model’s difficulty generalizing to longer contexts, rather than to the additional history becoming less informative. A larger context window therefore creates an opportunity to use more evidence; the model still has to use that evidence effectively.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Sparse histories and unseen surfaces
Cold-start members test whether the model can personalize without a rich interaction record. The presented cohorts include members with fewer than five interactions and members with fewer than 100. Sanjabi reports that 360Brew’s advantage over the production models grows as the number of interactions decreases. He interprets this as a benefit of the LLM’s world knowledge: the model can bring useful prior understanding to profiles and content when behavioral evidence is limited.
Sanjabi reports that, on four tasks whose surfaces were excluded from training, 360Brew matched or exceeded some production models trained specifically for those tasks. The relevant comparison is against specialized models with exposure to their own task, not against untrained baselines. These are model-quality comparisons; they do not establish online business gains or replacement of the production rankers. The proposed operational benefit is faster rollout: a new feature or surface could begin with the shared model instead of waiting for a dedicated recommendation system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compress gradually, with recovery between steps
Serving changes the objective. Firooz describes recommendation workloads requiring tens of thousands of queries per second and roughly 400–500 ms latency. Meeting that demand requires combining smaller models, sparsification and quantization rather than relying on model size alone.
The distillation process reduces capacity in stages. Firooz describes an illustrative progression from a large teacher through 8B, 3B and 1B students, with each student learning from the preceding model. His account uses both 250B and 150B for the starting teacher, so the exact teacher size in this serving recipe is inconsistent; the earlier Brew-XL description uses 150B. The stable result is the comparison between procedures: staged distillation worked better for the team than jumping directly from 150B to 1B. Starting small also performed poorly, which Firooz attributes to insufficient capacity and reasoning ability for the tasks.
Pruning follows the same gradual pattern. Transformer redundancy creates opportunities to remove attention heads, MLPs or layers; reducing activation and parameter precision is another compression option. But removing too much structure at once can damage quality. The team instead alternates a small pruning step with distillation, then repeats. Distillation provides a recovery stage before the next reduction.
In the presented pruning experiment, Firooz describes gradual pruning as approaching no information loss, while aggressive initial pruning caused up to a 1% reduction in model quality. The talk does not specify the quality metric or whether that percentage is relative or a percentage-point change. The practical comparison is between a sequence of small reductions with recovery and one large initial reduction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep enough precision to distinguish candidates
Quantization reduces the cost of activations and model parameters, but the team found that applying FP8 uniformly across every layer significantly hurt quality. Their solution is mixed precision, with special treatment for the output that becomes the ranking score.
| Model component | Precision choice | Purpose |
|---|---|---|
| Activations and parameters where suitable | FP8 | Reduce serving cost |
| Final LM head | FP32 | Preserve output-score resolution |
For this system, Firooz reports that using FP16, BF16 or FP8 for the LM head caused numerical distinctions to collapse, impaired calibration and made recommended items harder to distinguish. Ranking depends on differences between candidate probabilities, so preserving those differences can matter even when much of the model tolerates lower precision. The FP32 head is a system-specific finding, not a claim that every ranking model needs the same precision policy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Remove attention work the task does not need
Attention is another major serving cost. A recommendation task gives the implementation structure it can exploit: not every history item needs to attend to every other item. Rather than preserve all possible attention links, the model can use a sparse pattern tailored to the information needed for prediction.
The same principle applies to multi-item scoring. Instead of processing one candidate at a time, the system can score tens or hundreds together—Firooz gives 50 and 500 as examples. The crucial constraint is that the candidate items must not attend to one another. Putting them in the same request should not make one candidate’s prediction depend on the contents of the other candidates.
Across four or five successive releases, Firooz reports a 7× reduction in latency and a 30× increase in queries handled per GPU. These are combined gains from the serving optimizations, not measurements of sparsity alone. The talk does not specify the hardware, workload, batch and context settings, or latency percentile needed to reproduce the comparison. The chart captures the intended outcome: more work from each GPU while each query waits less.
The prepared presentation closes by pointing to technical reports and papers sharing lessons from the project. The audience questions then fill in several practical details: how generalization is checked, how multi-item scoring executes, and how history selection interacts with serving.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What transfers—and what is available out of the box?
An audience member asks the natural follow-up to the unseen-task results: after so much fine-tuning, how does the team know the model will work on the next five tasks? Firooz describes a suite of approximately 50–60 internal and external benchmarks. IFEval checks retained instruction following, while LinkedIn tasks excluded from training test transfer to new domains. Those checks address different capabilities: following instructions is not itself evidence of recommendation quality on a new surface. The answer supplies ongoing empirical tests, rather than a guarantee about future tasks.
The next question moves from LinkedIn’s infrastructure to a smaller website. An audience member introduces Microsoft’s NLWeb, then asks what a real-estate site with thousands of listings could use as an out-of-the-box recommender. Sanjabi says he is unfamiliar with NLWeb and does not evaluate it. He also does not identify a ready-made model for the listing-site problem. Making a foundation model useful beyond large companies is part of the project’s ambition, but the answer does not offer an immediately available solution. The audience member recommends investigating NLWeb, and Sanjabi agrees to look.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Multi-item scoring happens during prefill
Does scoring multiple items mean generating multiple answers through several decoding steps? Firooz clarifies that the system keeps the work in prefill. It does not use multi-step generation or speculative decoding for this operation. The candidate items are sequenced together in the input, and the attention pattern prevents them from attending to one another.
The team implemented custom attention-mask kernel work in SGLang and vLLM. Firooz describes query segments containing up to 500 candidates, with candidates attending to member history and profile information rather than to other candidates. These are the team’s serving integrations described in the talk, not a claim that a current default installation exposes the same behavior.
Using the earlier feed example, add a second candidate about office furniture. At the level of whole prompt regions, the relevant cross-region connections are:
Candidate query Profile + history Other candidate
Database-index post allowed blocked
Office-furniture post allowed blocked
Both candidates can use the same evidence about the engineer’s interests and past responses. Neither can use the other candidate as additional evidence. This is the separation the mask must preserve; the table abstracts away token-level attention within each region. No engagement outcome has been assigned—the operation organizes the evidence available when the model computes the scores.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose history without disrupting the cache
A member’s history can include job applications, job postings and many other entities. Fitting that history into context raises a selection problem before inference begins. Firooz describes experiments with three approaches:
- Retrieval: Use a RAG system to find historical items closest to the current query.
- Chronological ordering: Supply interactions according to their order in time.
- Time-based weighting: Experiment with weight decay over chronological history.
For most of the team’s applications, chronological ordering was sufficient. Firooz connects that result to recommendation’s preference for freshness: recent activity is often particularly useful evidence of what a member wants now.
Selection is not only about relevance or recency. The context also contains a distribution of positive and negative examples, and the team must decide how to balance them and how much history to include. Those are familiar ML engineering questions inside a new input format: converting interactions to text does not remove sampling decisions.
Sanjabi adds a serving constraint: a more sophisticated history-selection strategy must avoid breaking KV caching. Changing which historical items appear, or how they are ordered, can interfere with the cache reuse the serving system depends on. An alternative to chronological history therefore has to earn its place on both prediction quality and serving behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the whole experiment repeatable
The final question asks how the team manages so many combinations of model and serving changes. Firooz’s answer is concrete: experiment results are automatically pushed into an Excel sheet. A developer investigating quantization can change its parameters and trigger the rest of the flow end to end with a button. Automation makes it practical to compare variants repeatedly, rather than manually shepherding each experiment through disconnected stages.
Most of that automation was built in-house, but it integrates open-source tools including Lightning, vLLM and SGLang. The engineering emphasis is on making those components work well together and optimizing the entire flow. The development system is what lets the team repeatedly move from a quality improvement to compression, serving evaluation and a recorded result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original benchmark paper explaining reproducible checks for verifiable natural-language instructions.
Open-source tools and protocols for adding conversational interfaces to websites using structured content.
Further reading
Original research abstract describing the 150B model and offline evaluation across more than 30 prediction tasks. The arXiv record now carries a licensing-related withdrawal notice.
Read the complete timestamped transcript
- 0:00
[on-hold electronic music]
- 0:14
Hi, everyone. Uh, very excited to be here. And, uh, I'm Hamed, uh, this is Maziar, and, uh, today, uh, uh, we're gonna talk about our journey in leveraging large language models for personalization and ranking, uh, and our path to productionize such a large model for, uh, uh, for LinkedIn use cases.
- 0:35
Oops! Okay. Uh, recommendation ranking and personalization is, uh, deeply integrated our, our daily life. Uh, when you go to a feed to, uh, to read an article, when you're looking for a, for a job, when you are searching for something, when you're buying something online, the, the, uh, the backend, uh, powered by recommendation system tries to find
- 0:56
the, the, the best, uh, content or best entity based on your, uh, your interest and, and relevancy to, to your, uh, to, to you. Uh, however, this, uh, this system usually, um,
- 1:11
um, suffer from some, some challenges, especially they are, they are, they are being, being trained on a specific task. So they are disjoint, uh, optimized. Um, they are usually not leveraging...
- 1:24
For, for leveraging the, the most advanced architecture that are being rolled out one by one, which is very time-consuming and, uh, and unproductive. Um,
- 1:33
so what the question that we are asking is that, what if we have only one model to, to solve all the tasks, uh, at, at, uh, at the same time?
- 1:43
So the mission that we started was to build a large foundation, uh, model based on large language models that understand the, the holist- have a holistic understanding of the user, uh, journey on LinkedIn platform and can solve all the personalization tasks that, that LinkedIn has with just one model.
- 2:02
And in addition to that, we wanted this model to have three, uh, three other main characteristics. One, we want this model to have zero-shot capability so that when you have a new problem or new surface, instead of basically collecting the data, building a new recommendation, uh, ranking, uh, ranking model and, and, and putting into production, which is
- 2:20
a very time-consuming, uh, journey, we can basically leverage this model out of the box to solve your task. You just basically prompt the model and, uh, tell the model that this is the task that I want to solve, this is kind of recommendation, this is the, this is the entity, this is the user, and what do you
- 2:35
think about the relevancy between these two entities?
- 2:39
Uh, the second characteristic that we want to have this model to have is to leverage in-context learning as much as possible, so that for a cold start user's problem, for example, uh, we can leverage this model by just giving a very few examples or, or just by explaining what the user might be interested in, and the model
- 2:55
can solve that problem for the cold start users.
- 2:58
And the last one is following instruction. We want basically give our users and members the ability to, uh, to tell the model what they're interested in. Like, imagine that next time that you go to LinkedIn feed or, uh, uh, you can tell the model that these are the, these are the niche, these are my niche interests, and
- 3:16
these are the, the topics that I'm interested to, to explore. And the model is basically the recommendation system start finding the relevant information for you and recommend it to you.
- 3:27
Now, Maziar will talk about how we build this model, uh, and then I'll talk about how to serve this model.
- 3:32
Okay. So it's on. So let me talk a little bit about the, uh, the brewing part, the building of the model. So in order to make use of the LLMs, which is what I think most of you guys are here for, is that we need to convert all the information we have about the users and the interactions
- 3:50
and everything that they had into prompt. And this is what we call the magical promptification. So we take the, uh, all the information we have about the user history and their profiles and a lot of interactions that they have had, and we turn it into a prompt, something like the one on the right-hand side here.
- 4:05
So there is, as you can see, there is an instruction for the model to follow, for example, what we want the model to do in this case, so that we can actually generalize over the different instructions.
- 4:14
We give some information about the member profile, and we have some, uh, past, for example, interactions that they have had with the data that we have already shown to them.
- 4:22
And then the question comes in: What do you think the user is going to do with this data or this, this new piece of information or this new item that we are showing to you?
- 4:31
So that's basically how we formalize the problem in order to feed it into an LLM. So obviously, I mean, if you take one of the LLMs out of the box and try to solve this problem with, it's gonna work a little bit, but it's not gonna be perfect.
- 4:44
So in order to do that, we have to train the model. So this is actually the pipeline that, uh, we have for developing the model and making it productionized.
- 4:51
So as you can see, the left-hand side, we start with the open source model. Then we do some, uh, magic of upcycling to, uh, to basically so that we can actually control the size of the model and the throughput versus the quality of the model.
- 5:04
And then we have like a few, uh, blocks of training, uh, continuous pretraining, uh, fine-tuning and inf- uh, instruction fine-tuning and also alignment. And at this point, we have this, uh, large model, which is, we call Brew-XL, which you can think of it as a large model with one hundred and fifty billion parameters that does really, really
- 5:22
well. And we are, we have maximized the quality. But obviously, this model is not going to be able to serve online because as you, as you know, the recommendation systems are very, very, uh, QPS-hungry.
- 5:33
So from here, we go all the way down to try to distill the model, so maximize the efficiency, and we're gonna talk a little bit about that. But basically, we go all the way down to, let's say, 3B model, which is actually something that can be productionized.
- 5:45
But as you can see, there are so many s- uh, different boxes here. And in order to make sure that the, the, the development cycle is actually smooth, we had to do a lot of automation.
- 5:54
So one of the key lessons from here is that you build a lot of automation into this, uh, into these pipelines in order to make, uh, ma-make the fact that making these models is actually very complicated into much easier and more manageable, uh, situation.
- 6:09
Uh, one big question that might actually come up here is that why do you actually need the XL model? And in fact, we did a lot of experimentation to see if we can actually get away from, uh, with not having the XL model.
- 6:19
Unfortunately, that's not actually the case. You have to first go big and then go small. If you do, uh, try to train the model from scratch with a small model, it doesn't actually work that well.
- 6:28
So in this case, we did this, and we show that the distillation is actually something that is very important for, for the smaller models. But now let me tell you a little bit about the levers that you can use in order to improve these models over time.
- 6:40
This is actually something that's very important. I mean, if you look at all the literature, there is a lot about scaling laws, how these models actually scale with data, with compute, and, uh, with this and that.
- 6:50
So in this case, we have three different layers, and I'm gonna talk about. The first one is obviously the data scaling. So what if we have actually more and more data?
- 6:57
This is something that we... it comes up a lot in the, uh, in the recommendation systems. We actually have a lot of data. Depending on how much you actually log about the user behavior, you might have a lot of data that, uh, goes back to six months, one year, or whatever.
- 7:11
And in this graph, as, as, as you see, as we increase the amount of data, the performance of the model actually improves. And, uh, uh, we hope that we can actually improve the model even further with having more and more data feed, feed into it.
- 7:25
Um, the... another lever that you can actually pull in order to improve the quality of the model, especially the XL model, is, uh, to increase the size of the model.
- 7:33
And in this experiment, we actually did this experiment o-over Mixtral, uh, architecture. You can see if you go from seven B to eight by twenty-two B, the performance of the model actually increases and improves.
- 7:45
And finally, this is another thing that is kinda like, I think one of the take, take-home message from here would be that the context length actually matters a lot for these kinds of applications with the recommendation systems.
- 7:56
And the context length actually defines how much history from the user you can actually give to the model. So in this experiment, we actually show that if you increase the context length by, uh, feeding more history from the user to the model, you can actually improve the, uh, the performance of the model by feeding more and more
- 8:13
data to the model. As you can see, towards the end of this graph, the performance actually drops. Uh, we don't believe that this is because of the fact that the context is actually less informative.
- 8:23
The problem is that the models, I mean, at least the model that we were using in this experiment, doesn't generalize that well to the longer context. Actually, they are...
- 8:30
I mean, if you look at most of the literature, they tell that the, the, the performance of the model actually drops if you go beyond some context. So
- 8:39
I, uh, actually, I, I have to give it back to you.
- 8:41
Okay.
- 8:42
Okay. Uh, let's, uh, talk a little bit about the, uh, the results and see if we can actually, uh, deliver on some of the promises that I, that we had.
- 8:52
So one of the things that we promised was that we can actually improve the performance of the model or performance of the behav- the behavior of the system on cold start users.
- 9:01
In this case, we actually show the gap between our model and the production models, uh, uh, on the users that have few, uh, interactions, like for example, less than five interactions, less than hundred interactions, and, uh, so on.
- 9:16
And y- as you can see, the gap between the, uh, the, the 360Brew model and the production model actually grows, uh, as the number of interactions decreases. So this actually shows you that having the world knowledge, uh, that the model, uh, inserts into these, these systems actually improves the q- the quality, uh, of its, uh, predictions.
- 9:37
Finally, uh, uh, we, we were, uh, promising to give you some generalization to the new domains, meaning that the problems that model has never seen inside its training. And in this graph, as I show, these are four different tasks, and these tasks are completely out of domain, not...
- 9:56
no information about that surface the model has seen during the training. But as you can see, it can actually, uh, be on par or even beat, uh, some of the, um, the, the production models.
- 10:08
And just to say, these production models are specific for that specific task, so they have been trained on that task. So this is not actually a, the, the small feat, so it's actually something that's significant.
- 10:19
So as you can see, this gives the, uh, the people who are developing these, uh, uh, develop-developing these platforms to roll out, uh, features and roll out surfaces much more quickly because they can actually use these models to do, uh, to do, uh, recommendation for them.
- 10:35
And now I give it back to Hamed to talk about serving.
- 10:38
So let me walk you through that how can we production such a large model in an environment that requires a very high QPS and low latency. Many recommendation systems have tens of thousands of the, of the QPS, and they also require more...
- 10:53
less than a second, like a five hundred, four, four hundred millisecond latency, uh, at, at, at best. Um,
- 11:00
there are, there are three levers that we can, we can pull in, in order to make the model more efficient and improve the throughput and, and reduce the latency for these models: uh, sparsification, uh, going to the smaller model, and quantization.
- 11:17
Uh, as, uh, as, as Maziar, uh, uh, explained before, uh, smaller models definitely have a better throughput, but our recipe is that we need to go big and then go small.
- 11:28
If you go with a smaller model initially, it doesn't have enough capacity, it doesn't have enough reasoning power to, to, to solve the, the complicated task that we have.
- 11:37
So we go with a larger model, and then we start this, uh, two hundred and fifty billion parameter model, and then we start distilling it to the smaller model.
- 11:45
And one of the recipe here is that we need to do the distillation step by step, and that means that we go with the, for example, uh, eight B mo- uh, eight B model, then three B model, and then one B model.
- 11:55
So we slowly decrease the size of the model, and we, we, we, we, we distill over and over from the, from the, from the, uh, from the previous model.
- 12:04
Um, and that recipe shows to be much, much, much more effective rather than com- basically directly going from hundred and fifty billion parameter model to one B parameter model
- 12:14
Uh, same thing for pruning. Uh, so p-p-pruning is a matema-optimiza-mathematical optimization problem. You want to re-re-either reduce the, uh, reduce the, your s- your number of heads in the transformers.
- 12:26
You can reduce the number of M-MLPs. Overall, these transformer models tends, uh, uh, proven to be very, very redundant in terms of keeping the information. So we can start pruning and removing some of these layers, uh, or, or reduce basically the precision for, for, uh, for the, for the, for, for each of the activations and parameters.
- 12:46
Uh, however, uh, again, if, if you do the pruning, uh, uh, very aggressively at the beginning, your performance will significantly suffer. So the, the recipe here is also do the gradual pruning.
- 12:57
Uh, we, we do... we, what we, we do is that we start pruning the model, very s-small pruning to the model. We, we, we distill, uh, to the smaller model, and, uh, we do it over and over again.
- 13:08
More, more pruning, more distillation. More pruning, more distillation. And as you can see from this plot, uh, doing the gradual pruning, uh, has, uh, can, can be as effective as basically, um, no, no information loss.
- 13:22
Whereas if you just basically do aggressive pruning at the beginning, you can have up to one percent reduction in the, in the model quality.
- 13:31
Another lever is, is, is quantization, going to lower precision. Uh, we are leveraging FP8, uh, uh, for activation model parameters. However, uh, doing just FP8, uh, in all the layers, uh, hurts the performance of the model or the quality of the model significantly.
- 13:48
So now basically your, your tool would be to do mixed precision. And one of the important aspect when it comes to ranking recommendations and, and, uh, overall, uh, prediction tasks is you want the model, the, the prediction or the probability output of the model to have a very good precision.
- 14:03
So in, uh, the LM head at the end, end of the language model has to be in FP32. If you do it in FP16, BF16 or FP8, uh, what, what happens is that the numbers collapse, and you don't have a very good calibration on top of that, and you cannot, uh, distinguish between different i-item recommended.
- 14:22
Uh, last part is, uh, is sparsification. We can sparsify basically the, the, the attentions. The most expensive part of the transformers is, uh, is attention scores, and we can leverage sparsification.
- 14:34
Not every item needs to attend to every items. And when you know your task, when you know that this recommendation, these are the items that you want to, uh, in the history, you can sparsify and not have every item attend to each other.
- 14:46
And same, same goes with when you are recommending the items. Instead of recommending one item, you can recommend fifty item, five hundred item at the same time. But you wanna make sure that these items are not attending to each other, so you, you sparsify, uh, the, the attention scores, uh, uh, for the output and for the, for
- 15:02
the query. If you put everything together, uh, we can, we can see that basically this, this... we can, we can have a significant reduction in the latency. What we have done is that in the, in a, in four or five of our release, uh, uh, one release after the other, we were able to reduce the latency by
- 15:23
seven X. And at the same time increasing the throughput, which is basically the number of queries that we can handle by one GPU, uh, by thirty X. So we are improving basically the, the amount of the work, work that a GPU is doing.
- 15:35
At the same time, we are reducing the latency that each query is seeing.
- 15:43
Uh, these are some of basically technical report and, and, and, um, uh, papers that we published, uh, during our journey to share with the community basically our, our, our lesson learned.
- 15:54
Um, and that's the end of our talk. So we have some time also to answer some questions.
- 16:03
Thank you.
- 16:03
Thank you. [audience applauding]
- 16:07
Please come to the microphones, um, if you wanna ask a question. Yeah.
- 16:11
Yeah. Uh, thank you. Great talk. One question. How did you measure that it doesn't lose generalization power? Obviously, you've done a lot of fine-tuning, uh, and you, you mentioned it works for four or five tasks, uh, instead of task-specific models.
- 16:22
How do you know it's gonna work for the next five tasks?
- 16:24
That's a good question. So we have a lot of... I mean, the answer overall is having a very comprehensive benchmarking set. We have something around like, uh, fifty to sixty benchmarking.
- 16:34
Some of them are internal, some of them are external. For example, we leverage IFEVA to make sure that the model still follows a very good instruction. Um, and, uh, as Maziar mentioned, some of the tasks are not never been part of our training data, and that's how we are j- measuring basically the generalization to the new, new
- 16:50
domain within LinkedIn use cases, for example. Yeah.
- 16:54
Hi. Thank, thanks for the talk. Um, I'm wondering what a small, uh, listing website, uh, can use out of the box. Um, have you heard of NLWeb, which was launched recently by Microsoft?
- 17:06
Uh, if yes, what are your views on that as a recommendation system?
- 17:10
NLWeb? What?
- 17:11
NLWeb.
- 17:11
No, I haven't actually heard of it.
- 17:12
No. Okay. Okay, cool.
- 17:13
Sorry about that.
- 17:14
Anything you... for smaller ones, listing, let's say real estate listing website has like thousands of, uh, real estate listings. What are the out-of-the-box recommendation models that, uh, people can start using?
- 17:27
Uh, I mean, that's the... [chuckles] I, uh, I wish that, that such a model would exist. I, I don't really... I mean, that's why I think we started this work.
- 17:35
We try... we are trying to see if we can actually make it a foundation model so that you can actually solve those kinds of problems. I think there is a lot of potential for, for this to be able to serve a lot of the use cases that are beyond the bigger companies.
- 17:47
But definitely I don't know any of-
- 17:48
I think, uh, you should check out NLWeb one.
- 17:51
Okay. I'll look at that.
- 17:52
Yeah. Yeah. Thanks.
- 17:55
Okay.
- 17:55
Uh, thank you for the great talk. Um, uh, on the slide where you mentioned you have a multi, uh, item scoring-
- 18:02
Uh-huh
- 18:02
... uh, I'm curious, like, uh, what does that, uh, effectively mean? Does it mean that you need to do multi-step decoding, or it's just a one-step or just processing the logits for multiple items?
- 18:13
What does it ensure?
- 18:14
It's a multi-step. We don't wanna basically... We didn't want to go to the, uh, for example, complication of speculative decoding or-
- 18:21
Mm-hmm
- 18:21
... basically the decoding aspect. We wanted to have everything at the prefill. So what we did was that basically all the items are being sequenced or, or they're recommended items or potential candidates are sequenced together.
- 18:32
Mm-hmm.
- 18:32
But we also wanted to avoid them to attend to each other.
- 18:36
Mm.
- 18:37
So we leverage basically what we call it like a, a f- for the attention mask. Uh, and we, we develop a, a special care and actually in the SG rank, uh, and VLLM too, to be able to do that.
- 18:48
And now when you have, uh, up to five hundred, uh, items in, in your query segment-
- 18:53
Mm-hmm
- 18:53
... uh, those items don't attend to each other. They only attend to the historical user and, and, and user profile information.
- 19:00
Okay. Thank you.
- 19:03
Hey, great talk. Uh, so a user history means many things, right? So like there is all of the jobs that they've applied to or, and the job postings. There are so many entities and so on.
- 19:13
Uh, the, the context of the model can get quite large. Uh, how did you manage that? Did you compress it or, uh, were there parts that you focused on?
- 19:22
Yeah. So we, we actually, uh, experimented a lot of things. Uh, we experimented with, with RAG system, so that basically when we have a query, we try to figure out what are the most closest items in the your history to bring it up.
- 19:34
Uh, we also, uh, experimented with chronical orders-
- 19:37
Mm-hmm
- 19:38
... and some sort of weight decay on the chronical orders.
- 19:41
Mm-hmm.
- 19:41
It turns out that for majority of application that we have, actually chronical order-
- 19:45
Mm-hmm
- 19:45
... is good enough, and that kind of makes sense because recommendation systems are very biased to the freshness.
- 19:50
Yeah.
- 19:51
So the, the more recent user activity helps.
- 19:54
Mm-hmm.
- 19:54
One of the biggest challenge is actually the... This is more, now become more like a traditional LM problem. How do you balance the distribution of your positive and negative within the context?
- 20:04
Mm-hmm.
- 20:04
And I think that's become something that more like a ML engineering e-effort to figure out, "Okay, do I want more positive, more negative? Like, how much, how much information I need to put in the context?"
- 20:14
Got it.
- 20:15
Yeah.
- 20:15
So j-just one-
- 20:16
I can add one more thing to this.
- 20:18
Sure.
- 20:18
This, there's also another complication. When you go to the serving of these models, you don't want to break the KV caching or something that you're using in the serving.
- 20:25
So it's going to be a little bit more complicated, more cumbersome to do something that's, uh, smarter than just putting the chronological order. So that's something that needs to be designed.
- 20:33
So it's not something that's very obvious.
- 20:35
Absolutely. Uh, one more question. Uh, you guys did so many experiments, tried out so many things. Uh, how's your entire system set up? Because I'm assuming that you, you say quantization, but you must have tried different forms of quantization and whatnot.
- 20:47
H-How do you set up the system in such a way that you can try out multiple experiments and see what works best? Can you talk a bit about that?
- 20:55
Uh, yes. So Maziar touched a bit on that one. I think the, the, the, the one thing that we, we, we hold a very high bar for the one was automation.
- 21:02
Mm-hmm.
- 21:02
So our system is very automated, uh, to the extent that when you are running experimentation, actually the result of the experiment is, is being pushed automatically into the Excel sheet.
- 21:12
And now when you have a such automated system, now basically the developers are very efficient in terms of like, "I just want to figure out different quantization." So you just change the quantization parameters, and everything else just by clicking a button happens end to end.
- 21:24
Uh, so I think, uh, automation is the key if you want to basically really, uh, optimize for these models.
- 21:30
S-So did you build all of that automation in-house, or did you-
- 21:33
Yes, most of them. Well, we leveraged, for example, Lightening, VLLM, SG like.
- 21:37
Mm-hmm.
- 21:37
Uh, we leveraged basically a lot of open source tools, but we make sure that they are, they are integrated very well with each other and, uh, and optimize basically the entire flow.
- 21:47
Cool. Thank you.
- 21:48
Thank you for, uh, thank you again, Hamed, Maziar. [clapping]
- 21:52
Thank you. [upbeat music]