360Brew: LLM-based Personalized Ranking and Recommendation — Hamed Firooz and Maziar Sanjabi, LinkedIn AI

Talk transcript

1. 0:00

   [on-hold electronic music]

2. 0:14

   Hi, everyone. Uh, very excited to be here. And, uh, I'm Hamed, uh, this is Maziar, and, uh, today, uh, uh, we're gonna talk about our journey in leveraging large language models for personalization and ranking, uh, and our path to productionize such a large model for, uh, uh, for LinkedIn use cases.

3. 0:35

   Oops! Okay. Uh, recommendation ranking and personalization is, uh, deeply integrated our, our daily life. Uh, when you go to a feed to, uh, to read an article, when you're looking for a, for a job, when you are searching for something, when you're buying something online, the, the, uh, the backend, uh, powered by recommendation system tries to find

4. 0:56

   the, the, the best, uh, content or best entity based on your, uh, your interest and, and relevancy to, to your, uh, to, to you. Uh, however, this, uh, this system usually, um,

5. 1:11

   um, suffer from some, some challenges, especially they are, they are, they are being, being trained on a specific task. So they are disjoint, uh, optimized. Um, they are usually not leveraging...

6. 1:24

   For, for leveraging the, the most advanced architecture that are being rolled out one by one, which is very time-consuming and, uh, and unproductive. Um,

7. 1:33

   so what the question that we are asking is that, what if we have only one model to, to solve all the tasks, uh, at, at, uh, at the same time?

8. 1:43

   So the mission that we started was to build a large foundation, uh, model based on large language models that understand the, the holist- have a holistic understanding of the user, uh, journey on LinkedIn platform and can solve all the personalization tasks that, that LinkedIn has with just one model.

9. 2:02

   And in addition to that, we wanted this model to have three, uh, three other main characteristics. One, we want this model to have zero-shot capability so that when you have a new problem or new surface, instead of basically collecting the data, building a new recommendation, uh, ranking, uh, ranking model and, and, and putting into production, which is

10. 2:20

    a very time-consuming, uh, journey, we can basically leverage this model out of the box to solve your task. You just basically prompt the model and, uh, tell the model that this is the task that I want to solve, this is kind of recommendation, this is the, this is the entity, this is the user, and what do you

11. 2:35

    think about the relevancy between these two entities?

12. 2:39

    Uh, the second characteristic that we want to have this model to have is to leverage in-context learning as much as possible, so that for a cold start user's problem, for example, uh, we can leverage this model by just giving a very few examples or, or just by explaining what the user might be interested in, and the model

13. 2:55

    can solve that problem for the cold start users.

14. 2:58

    And the last one is following instruction. We want basically give our users and members the ability to, uh, to tell the model what they're interested in. Like, imagine that next time that you go to LinkedIn feed or, uh, uh, you can tell the model that these are the, these are the niche, these are my niche interests, and

15. 3:16

    these are the, the topics that I'm interested to, to explore. And the model is basically the recommendation system start finding the relevant information for you and recommend it to you.

16. 3:27

    Now, Maziar will talk about how we build this model, uh, and then I'll talk about how to serve this model.

17. 3:32

    Okay. So it's on. So let me talk a little bit about the, uh, the brewing part, the building of the model. So in order to make use of the LLMs, which is what I think most of you guys are here for, is that we need to convert all the information we have about the users and the interactions

18. 3:50

    and everything that they had into prompt. And this is what we call the magical promptification. So we take the, uh, all the information we have about the user history and their profiles and a lot of interactions that they have had, and we turn it into a prompt, something like the one on the right-hand side here.

19. 4:05

    So there is, as you can see, there is an instruction for the model to follow, for example, what we want the model to do in this case, so that we can actually generalize over the different instructions.

20. 4:14

    We give some information about the member profile, and we have some, uh, past, for example, interactions that they have had with the data that we have already shown to them.

21. 4:22

    And then the question comes in: What do you think the user is going to do with this data or this, this new piece of information or this new item that we are showing to you?

22. 4:31

    So that's basically how we formalize the problem in order to feed it into an LLM. So obviously, I mean, if you take one of the LLMs out of the box and try to solve this problem with, it's gonna work a little bit, but it's not gonna be perfect.

23. 4:44

    So in order to do that, we have to train the model. So this is actually the pipeline that, uh, we have for developing the model and making it productionized.

24. 4:51

    So as you can see, the left-hand side, we start with the open source model. Then we do some, uh, magic of upcycling to, uh, to basically so that we can actually control the size of the model and the throughput versus the quality of the model.

25. 5:04

    And then we have like a few, uh, blocks of training, uh, continuous pretraining, uh, fine-tuning and inf- uh, instruction fine-tuning and also alignment. And at this point, we have this, uh, large model, which is, we call Brew-XL, which you can think of it as a large model with one hundred and fifty billion parameters that does really, really

26. 5:22

    well. And we are, we have maximized the quality. But obviously, this model is not going to be able to serve online because as you, as you know, the recommendation systems are very, very, uh, QPS-hungry.

27. 5:33

    So from here, we go all the way down to try to distill the model, so maximize the efficiency, and we're gonna talk a little bit about that. But basically, we go all the way down to, let's say, 3B model, which is actually something that can be productionized.

28. 5:45

    But as you can see, there are so many s- uh, different boxes here. And in order to make sure that the, the, the development cycle is actually smooth, we had to do a lot of automation.

29. 5:54

    So one of the key lessons from here is that you build a lot of automation into this, uh, into these pipelines in order to make, uh, ma-make the fact that making these models is actually very complicated into much easier and more manageable, uh, situation.

30. 6:09

    Uh, one big question that might actually come up here is that why do you actually need the XL model? And in fact, we did a lot of experimentation to see if we can actually get away from, uh, with not having the XL model.

31. 6:19

    Unfortunately, that's not actually the case. You have to first go big and then go small. If you do, uh, try to train the model from scratch with a small model, it doesn't actually work that well.

32. 6:28

    So in this case, we did this, and we show that the distillation is actually something that is very important for, for the smaller models. But now let me tell you a little bit about the levers that you can use in order to improve these models over time.

33. 6:40

    This is actually something that's very important. I mean, if you look at all the literature, there is a lot about scaling laws, how these models actually scale with data, with compute, and, uh, with this and that.

34. 6:50

    So in this case, we have three different layers, and I'm gonna talk about. The first one is obviously the data scaling. So what if we have actually more and more data?

35. 6:57

    This is something that we... it comes up a lot in the, uh, in the recommendation systems. We actually have a lot of data. Depending on how much you actually log about the user behavior, you might have a lot of data that, uh, goes back to six months, one year, or whatever.

36. 7:11

    And in this graph, as, as, as you see, as we increase the amount of data, the performance of the model actually improves. And, uh, uh, we hope that we can actually improve the model even further with having more and more data feed, feed into it.

37. 7:25

    Um, the... another lever that you can actually pull in order to improve the quality of the model, especially the XL model, is, uh, to increase the size of the model.

38. 7:33

    And in this experiment, we actually did this experiment o-over Mixtral, uh, architecture. You can see if you go from seven B to eight by twenty-two B, the performance of the model actually increases and improves.

39. 7:45

    And finally, this is another thing that is kinda like, I think one of the take, take-home message from here would be that the context length actually matters a lot for these kinds of applications with the recommendation systems.

40. 7:56

    And the context length actually defines how much history from the user you can actually give to the model. So in this experiment, we actually show that if you increase the context length by, uh, feeding more history from the user to the model, you can actually improve the, uh, the performance of the model by feeding more and more

41. 8:13

    data to the model. As you can see, towards the end of this graph, the performance actually drops. Uh, we don't believe that this is because of the fact that the context is actually less informative.

42. 8:23

    The problem is that the models, I mean, at least the model that we were using in this experiment, doesn't generalize that well to the longer context. Actually, they are...

43. 8:30

    I mean, if you look at most of the literature, they tell that the, the, the performance of the model actually drops if you go beyond some context. So

44. 8:39

    I, uh, actually, I, I have to give it back to you.

45. 8:41

    Okay.

46. 8:42

    Okay. Uh, let's, uh, talk a little bit about the, uh, the results and see if we can actually, uh, deliver on some of the promises that I, that we had.

47. 8:52

    So one of the things that we promised was that we can actually improve the performance of the model or performance of the behav- the behavior of the system on cold start users.

48. 9:01

    In this case, we actually show the gap between our model and the production models, uh, uh, on the users that have few, uh, interactions, like for example, less than five interactions, less than hundred interactions, and, uh, so on.

49. 9:16

    And y- as you can see, the gap between the, uh, the, the 360Brew model and the production model actually grows, uh, as the number of interactions decreases. So this actually shows you that having the world knowledge, uh, that the model, uh, inserts into these, these systems actually improves the q- the quality, uh, of its, uh, predictions.

50. 9:37

    Finally, uh, uh, we, we were, uh, promising to give you some generalization to the new domains, meaning that the problems that model has never seen inside its training. And in this graph, as I show, these are four different tasks, and these tasks are completely out of domain, not...

51. 9:56

    no information about that surface the model has seen during the training. But as you can see, it can actually, uh, be on par or even beat, uh, some of the, um, the, the production models.

52. 10:08

    And just to say, these production models are specific for that specific task, so they have been trained on that task. So this is not actually a, the, the small feat, so it's actually something that's significant.

53. 10:19

    So as you can see, this gives the, uh, the people who are developing these, uh, uh, develop-developing these platforms to roll out, uh, features and roll out surfaces much more quickly because they can actually use these models to do, uh, to do, uh, recommendation for them.

54. 10:35

    And now I give it back to Hamed to talk about serving.

55. 10:38

    So let me walk you through that how can we production such a large model in an environment that requires a very high QPS and low latency. Many recommendation systems have tens of thousands of the, of the QPS, and they also require more...

56. 10:53

    less than a second, like a five hundred, four, four hundred millisecond latency, uh, at, at, at best. Um,

57. 11:00

    there are, there are three levers that we can, we can pull in, in order to make the model more efficient and improve the throughput and, and reduce the latency for these models: uh, sparsification, uh, going to the smaller model, and quantization.

58. 11:17

    Uh, as, uh, as, as Maziar, uh, uh, explained before, uh, smaller models definitely have a better throughput, but our recipe is that we need to go big and then go small.

59. 11:28

    If you go with a smaller model initially, it doesn't have enough capacity, it doesn't have enough reasoning power to, to, to solve the, the complicated task that we have.

60. 11:37

    So we go with a larger model, and then we start this, uh, two hundred and fifty billion parameter model, and then we start distilling it to the smaller model.

61. 11:45

    And one of the recipe here is that we need to do the distillation step by step, and that means that we go with the, for example, uh, eight B mo- uh, eight B model, then three B model, and then one B model.

62. 11:55

    So we slowly decrease the size of the model, and we, we, we, we, we distill over and over from the, from the, from the, uh, from the previous model.

63. 12:04

    Um, and that recipe shows to be much, much, much more effective rather than com- basically directly going from hundred and fifty billion parameter model to one B parameter model

64. 12:14

    Uh, same thing for pruning. Uh, so p-p-pruning is a matema-optimiza-mathematical optimization problem. You want to re-re-either reduce the, uh, reduce the, your s- your number of heads in the transformers.

65. 12:26

    You can reduce the number of M-MLPs. Overall, these transformer models tends, uh, uh, proven to be very, very redundant in terms of keeping the information. So we can start pruning and removing some of these layers, uh, or, or reduce basically the precision for, for, uh, for the, for the, for, for each of the activations and parameters.

66. 12:46

    Uh, however, uh, again, if, if you do the pruning, uh, uh, very aggressively at the beginning, your performance will significantly suffer. So the, the recipe here is also do the gradual pruning.

67. 12:57

    Uh, we, we do... we, what we, we do is that we start pruning the model, very s-small pruning to the model. We, we, we distill, uh, to the smaller model, and, uh, we do it over and over again.

68. 13:08

    More, more pruning, more distillation. More pruning, more distillation. And as you can see from this plot, uh, doing the gradual pruning, uh, has, uh, can, can be as effective as basically, um, no, no information loss.

69. 13:22

    Whereas if you just basically do aggressive pruning at the beginning, you can have up to one percent reduction in the, in the model quality.

70. 13:31

    Another lever is, is, is quantization, going to lower precision. Uh, we are leveraging FP8, uh, uh, for activation model parameters. However, uh, doing just FP8, uh, in all the layers, uh, hurts the performance of the model or the quality of the model significantly.

71. 13:48

    So now basically your, your tool would be to do mixed precision. And one of the important aspect when it comes to ranking recommendations and, and, uh, overall, uh, prediction tasks is you want the model, the, the prediction or the probability output of the model to have a very good precision.

72. 14:03

    So in, uh, the LM head at the end, end of the language model has to be in FP32. If you do it in FP16, BF16 or FP8, uh, what, what happens is that the numbers collapse, and you don't have a very good calibration on top of that, and you cannot, uh, distinguish between different i-item recommended.

73. 14:22

    Uh, last part is, uh, is sparsification. We can sparsify basically the, the, the attentions. The most expensive part of the transformers is, uh, is attention scores, and we can leverage sparsification.

74. 14:34

    Not every item needs to attend to every items. And when you know your task, when you know that this recommendation, these are the items that you want to, uh, in the history, you can sparsify and not have every item attend to each other.

75. 14:46

    And same, same goes with when you are recommending the items. Instead of recommending one item, you can recommend fifty item, five hundred item at the same time. But you wanna make sure that these items are not attending to each other, so you, you sparsify, uh, the, the attention scores, uh, uh, for the output and for the, for

76. 15:02

    the query. If you put everything together, uh, we can, we can see that basically this, this... we can, we can have a significant reduction in the latency. What we have done is that in the, in a, in four or five of our release, uh, uh, one release after the other, we were able to reduce the latency by

77. 15:23

    seven X. And at the same time increasing the throughput, which is basically the number of queries that we can handle by one GPU, uh, by thirty X. So we are improving basically the, the amount of the work, work that a GPU is doing.

78. 15:35

    At the same time, we are reducing the latency that each query is seeing.

79. 15:43

    Uh, these are some of basically technical report and, and, and, um, uh, papers that we published, uh, during our journey to share with the community basically our, our, our lesson learned.

80. 15:54

    Um, and that's the end of our talk. So we have some time also to answer some questions.

81. 16:03

    Thank you.

82. 16:03

    Thank you. [audience applauding]

83. 16:07

    Please come to the microphones, um, if you wanna ask a question. Yeah.

84. 16:11

    Yeah. Uh, thank you. Great talk. One question. How did you measure that it doesn't lose generalization power? Obviously, you've done a lot of fine-tuning, uh, and you, you mentioned it works for four or five tasks, uh, instead of task-specific models.

85. 16:22

    How do you know it's gonna work for the next five tasks?

86. 16:24

    That's a good question. So we have a lot of... I mean, the answer overall is having a very comprehensive benchmarking set. We have something around like, uh, fifty to sixty benchmarking.

87. 16:34

    Some of them are internal, some of them are external. For example, we leverage IFEVA to make sure that the model still follows a very good instruction. Um, and, uh, as Maziar mentioned, some of the tasks are not never been part of our training data, and that's how we are j- measuring basically the generalization to the new, new

88. 16:50

    domain within LinkedIn use cases, for example. Yeah.

89. 16:54

    Hi. Thank, thanks for the talk. Um, I'm wondering what a small, uh, listing website, uh, can use out of the box. Um, have you heard of NLWeb, which was launched recently by Microsoft?

90. 17:06

    Uh, if yes, what are your views on that as a recommendation system?

91. 17:10

    NLWeb? What?

92. 17:11

    NLWeb.

93. 17:11

    No, I haven't actually heard of it.

94. 17:12

    No. Okay. Okay, cool.

95. 17:13

    Sorry about that.

96. 17:14

    Anything you... for smaller ones, listing, let's say real estate listing website has like thousands of, uh, real estate listings. What are the out-of-the-box recommendation models that, uh, people can start using?

97. 17:27

    Uh, I mean, that's the... [chuckles] I, uh, I wish that, that such a model would exist. I, I don't really... I mean, that's why I think we started this work.

98. 17:35

    We try... we are trying to see if we can actually make it a foundation model so that you can actually solve those kinds of problems. I think there is a lot of potential for, for this to be able to serve a lot of the use cases that are beyond the bigger companies.

99. 17:47

    But definitely I don't know any of-

100. 17:48

     I think, uh, you should check out NLWeb one.

101. 17:51

     Okay. I'll look at that.

102. 17:52

     Yeah. Yeah. Thanks.

103. 17:55

     Okay.

104. 17:55

     Uh, thank you for the great talk. Um, uh, on the slide where you mentioned you have a multi, uh, item scoring-

105. 18:02

     Uh-huh

106. 18:02

     ... uh, I'm curious, like, uh, what does that, uh, effectively mean? Does it mean that you need to do multi-step decoding, or it's just a one-step or just processing the logits for multiple items?

107. 18:13

     What does it ensure?

108. 18:14

     It's a multi-step. We don't wanna basically... We didn't want to go to the, uh, for example, complication of speculative decoding or-

109. 18:21

     Mm-hmm

110. 18:21

     ... basically the decoding aspect. We wanted to have everything at the prefill. So what we did was that basically all the items are being sequenced or, or they're recommended items or potential candidates are sequenced together.

111. 18:32

     Mm-hmm.

112. 18:32

     But we also wanted to avoid them to attend to each other.

113. 18:36

     Mm.

114. 18:37

     So we leverage basically what we call it like a, a f- for the attention mask. Uh, and we, we develop a, a special care and actually in the SG rank, uh, and VLLM too, to be able to do that.

115. 18:48

     And now when you have, uh, up to five hundred, uh, items in, in your query segment-

116. 18:53

     Mm-hmm

117. 18:53

     ... uh, those items don't attend to each other. They only attend to the historical user and, and, and user profile information.

118. 19:00

     Okay. Thank you.

119. 19:03

     Hey, great talk. Uh, so a user history means many things, right? So like there is all of the jobs that they've applied to or, and the job postings. There are so many entities and so on.

120. 19:13

     Uh, the, the context of the model can get quite large. Uh, how did you manage that? Did you compress it or, uh, were there parts that you focused on?

121. 19:22

     Yeah. So we, we actually, uh, experimented a lot of things. Uh, we experimented with, with RAG system, so that basically when we have a query, we try to figure out what are the most closest items in the your history to bring it up.

122. 19:34

     Uh, we also, uh, experimented with chronical orders-

123. 19:37

     Mm-hmm

124. 19:38

     ... and some sort of weight decay on the chronical orders.

125. 19:41

     Mm-hmm.

126. 19:41

     It turns out that for majority of application that we have, actually chronical order-

127. 19:45

     Mm-hmm

128. 19:45

     ... is good enough, and that kind of makes sense because recommendation systems are very biased to the freshness.

129. 19:50

     Yeah.

130. 19:51

     So the, the more recent user activity helps.

131. 19:54

     Mm-hmm.

132. 19:54

     One of the biggest challenge is actually the... This is more, now become more like a traditional LM problem. How do you balance the distribution of your positive and negative within the context?

133. 20:04

     Mm-hmm.

134. 20:04

     And I think that's become something that more like a ML engineering e-effort to figure out, "Okay, do I want more positive, more negative? Like, how much, how much information I need to put in the context?"

135. 20:14

     Got it.

136. 20:15

     Yeah.

137. 20:15

     So j-just one-

138. 20:16

     I can add one more thing to this.

139. 20:18

     Sure.

140. 20:18

     This, there's also another complication. When you go to the serving of these models, you don't want to break the KV caching or something that you're using in the serving.

141. 20:25

     So it's going to be a little bit more complicated, more cumbersome to do something that's, uh, smarter than just putting the chronological order. So that's something that needs to be designed.

142. 20:33

     So it's not something that's very obvious.

143. 20:35

     Absolutely. Uh, one more question. Uh, you guys did so many experiments, tried out so many things. Uh, how's your entire system set up? Because I'm assuming that you, you say quantization, but you must have tried different forms of quantization and whatnot.

144. 20:47

     H-How do you set up the system in such a way that you can try out multiple experiments and see what works best? Can you talk a bit about that?

145. 20:55

     Uh, yes. So Maziar touched a bit on that one. I think the, the, the, the one thing that we, we, we hold a very high bar for the one was automation.

146. 21:02

     Mm-hmm.

147. 21:02

     So our system is very automated, uh, to the extent that when you are running experimentation, actually the result of the experiment is, is being pushed automatically into the Excel sheet.

148. 21:12

     And now when you have a such automated system, now basically the developers are very efficient in terms of like, "I just want to figure out different quantization." So you just change the quantization parameters, and everything else just by clicking a button happens end to end.

149. 21:24

     Uh, so I think, uh, automation is the key if you want to basically really, uh, optimize for these models.

150. 21:30

     S-So did you build all of that automation in-house, or did you-

151. 21:33

     Yes, most of them. Well, we leveraged, for example, Lightening, VLLM, SG like.

152. 21:37

     Mm-hmm.

153. 21:37

     Uh, we leveraged basically a lot of open source tools, but we make sure that they are, they are integrated very well with each other and, uh, and optimize basically the entire flow.

154. 21:47

     Cool. Thank you.

155. 21:48

     Thank you for, uh, thank you again, Hamed, Maziar. [clapping]

156. 21:52

     Thank you. [upbeat music]