Low Level Technicals of LLMs

Talk transcript

1. 0:00

   [upbeat music] Welcome to the AI Engineers World Fair.

2. 0:16

   Um, this is the first workshop. Um, there's a few other running, but, like, thanks for coming. Um, we just arrived from, like, Australia with my brother. Um, I think he's over there somewhere, but yes, we just came here.

3. 0:27

   Um, yes, we didn't know a lot of stuff [laughs] about SF and I think maybe the US is a bit different from Australia. Um, but yeah, we're very excited to be here.

4. 0:35

   So like, we're gonna stay here for like a few months, so if you wanna meet up, you know, you can just hit me up, email or Twitter or wherever.

5. 0:41

   Um, so today I'm gonna to be talking about low-level technicals of language models. Um, yes, yes, I'm Daniel. Um, [laughs] but so we do have a website called Unsloth.ai if you wanna look, like look that up.

6. 0:53

   Like, um, there's like cute sloths and stuff. So my brother designed that. We'll be using [REDACTED:generic_id] TinyURLs. Oh, did it... Okay, it's still working. Yeah, so we'll be using [REDACTED:generic_id] TinyURLs.

7. 1:03

   Now t- so the first [REDACTED:generic_id] is... Oh, wait, I'll sh- yeah. So the slides are at tinyurl.com/unsloth. Um, hopefully that works. Um, and there's also Q&A, so like I'll be monitoring Q&A.

8. 1:14

   You can type any question that you like, and I will be answering questions as we go. Um, and that is at, uh, tinyurl.com/unslothqa. Um, so if those [REDACTED:generic_id] work...

9. 1:25

   So there'll be like, on the bottom if, you know, anyone doesn't get this, like there's on the very bottom of the footer there'll be like, we'll reshow these links.

10. 1:34

    Okay. Just, just, has, does it work? Yes? Okay. Good, good, good. Okay. Yeah, so you might know me from my tweets. Um, so Gemma like kind of like released an open source model a few months ago, and we just found a few like issues and bugs for like different implementations.

11. 1:55

    Um, so like for example, the first tweet that we ever did was about some sort of like the approximate Gelu, um, bug issue. And so like multiple implementations of Gemma that had different implementations.

12. 2:08

    Some of them use exact Gelu, some of them use approximate, and like, so which [REDACTED:generic_id] is correct? And so that's the question. Um, and so like we just tweeted about this, and that was like our first issue that we found.

13. 2:19

    We thought this was just like [REDACTED:generic_id] issue, but actually there were like many issues. Um, and so like we found more bugs. Um, and so I'm assuming maybe you know me from this.

14. 2:28

    Um, we did get partially recognizable for our [laughs] Gemma bug fixes. So, um, yeah. So w- today we'll be showing you how you can actually find these bugs and issues in language models and how you can actually like analyze and do this yourself, um, without, you know, um, us just doing it [laughs] manually ourselves.

15. 2:46

    And hopefully this can be like a open source project where everyone can find these issues automatically and help us to solve these issues. Um, I always thought about like, can we like automate this?

16. 2:55

    I don't think this can be automated. Um, there are actually many issues with these implementations, and it's not just Gemma. Um, for example, like, you know, we also analyzed Groq, um, and there's like some weird things in their code, like they're scaled by, um, [REDACTED:password] times 10 H, X over [REDACTED:password].

17. 3:13

    It's just a s- it's just a clamping mechanism. Um, you can see I also make mistakes sometimes. Um, I like, you know, said it's d- division and not multiplication, so sometimes I misread the code.

18. 3:23

    Um, that's because like, you know, I spend... When the code gets released, I quickly try to analyze them and sometimes I mistakenly say stuff, um, so I have to like lodge, you know, showcase corrections.

19. 3:32

    So yes, I am still human. Um, but yes, like, you know, we analyze lots of models and stuff like this, and hopefully by the end of these workshops... Well, actually, this work- Okay, it's actually not [REDACTED:generic_id] workshop, it's going to be like multiple things in [REDACTED:generic_id].

20. 3:45

    So like I decided to like talk about three things, but I, I'll tell you about that later. Um, hopefully you guys can like analyze and learn about how to do, like find bugs and stuff like that by the end of today.

21. 3:56

    Um, so another [REDACTED:generic_id] I did recently was like, you know, NVIDIA's Nemotron. I don't know if you saw this, but NVIDIA released a 340 billion parameter model, which is extremely large.

22. 4:07

    I'm assuming this is in, like either in preparation, like they have to release this earlier before Llama 405 billion [laughs], right? So they have to do this quickly. Um, and but there are like some weird interesting things like, you know, they use the squared value and not the normal SwiGLU and the other types.

23. 4:24

    Um, so that was very interesting. They were actually the first... Well, actually not the first, but like the first big model trained to be using these n- like other activation functions.

24. 4:32

    And there's als- other like other weird quirks and stuff like that. Um, and hopefully you'll be also able to like analyze, like, you know, whenever the code comes out, just read it and you'll get it.

25. 4:41

    Um, it does take some practice. Okay, like I take like... Okay, the first time when I read this code, like it took me like many, many days to read through like all these, um, architectures and understand exactly what they are.

26. 4:52

    But now it takes like 10 minutes, so like I'm sure you can like just... The code comes and just read it. Um, that's the whole goal of today.

27. 5:00

    And also language models, if you don't know, they're not just like issues and bugs and analysis of these architectures. The tokenizer is a totally separate beast from language models.

28. 5:10

    Right? Tokenization is like extreme- extremely annoying. Um, so like I also tweeted about like, you know, there's like different types of tokenization issues, like Mistral, Llama, Mixtral, like all these different types of like variants of Mistral from the Mistral team.

29. 5:24

    You know, they have like different tokenizations if you didn't notice. Um, like if you... So the smiley face, the sun smiley face is a space and if you actually tokenize them, um, depending on the model, you have different results.

30. 5:38

    And the question is which [REDACTED:generic_id] is correct? Um, unfortunately I do not know. Um, I did ask the Hugging Face team for this, um, and according to them some of them are correct and some of them are just because the Mistral team forgot to update the model to the fast tokenization variant.

31. 5:52

    Um, we'll be talking about this later as well. Um, but you can see even before you can train or run the model, the tokenizer is broken. So what are you going to do?

32. 6:02

    Um, it's a multi-pronged problem So we don't just do language models. Um, you know, like we don't ... You know, our experience is, you know, a bit broader than that.

33. 6:13

    So we actually, like, I ac- I used to actually major in maths and computer science so, um, yes, very fun. Well, actually I kind of did very badly in maths.

34. 6:22

    But anyways, um, yes, very, very fun. So, like, SVD, I don't know if anyone ... Has anyone done normal machine learning here? Oh, yes. Very good. There is a few people.

35. 6:32

    Um, so, like, the SVD p- I'm assuming, like, most people know PCA, yes? Principal component analysis. Yes. Okay, very good. Data visualization. It's a very powerful technique, more people should know about it.

36. 6:43

    Um, SVD. Okay, I don't, I don't know if people know about SVD or, like ... Okay, yeah. So it's a bit less well-known. I'm actually a bit confused why people don't know SVD.

37. 6:51

    It's actually the algorithm that's [REDACTED:generic_id] of the most important algorithms in all of, like, maths and computer science. Um, it literally underpins, like, many applications, um, and it is extremely important.

38. 7:03

    Um, maybe we'll talk about that, but yes, I'm a huge proponent of, like, telling people to learn more about SVD, so please do the singular value decomposition. That's, like, the must, must, must, must, must.

39. 7:13

    Okay? Like, that's the most important algorithm. It's because it's, like, [REDACTED:generic_id] algorithm, it can, like, spawn many other algorithms and it's, like, can be used for many purposes. Um, there's also, like, the QR decomposition.

40. 7:23

    Okay, probably, shoot, okay, probably don't ... No [REDACTED:generic_id] knows. The LU, Cholesky, like, there's a lot. Um, randomized SVD. Yes, that's extremely important as well. Um, and yeah, so we don't just do language models.

41. 7:35

    You can ask me any questions about, you know, stuff, um, maths or computer science. You, you have a question or ...

42. 7:42

    Real quick. Um, so do you think, um, so for the Nemotron, uh, 34B, um, is it a unique architecture for ... 'Cause you can only use Nemo loader right now to load and train-

43. 7:55

    Mm

44. 7:55

    ... and, you know, do the ... I think the data's the most valuable part, but we are attempting to try to convert it to a Hugging Face, uh, transformer safe tensors.

45. 8:04

    So, but we've had issues because we don't have the modeling file. So, um, I was wondering, do you think that it's similar to ... So at the same day they uploaded a 70B of Llama 3 that's Nemotron as well.

46. 8:18

    Mm.

47. 8:18

    Do you think that we can get some clues of to, how to build a Hugging Face implementation of the-

48. 8:24

    Yes

49. 8:25

    ... uh, from there?

50. 8:26

    So the question was for Nemotron, the code was not released for the actual inference and mod- training. Uh, you have to go through the Nemo training framework from NVIDIA, um-

51. 8:34

    Yeah. Well, but what I mean is that I, I can dump the weight. The weights are-

52. 8:38

    Yes, but the code, yeah.

53. 8:40

    Yeah.

54. 8:40

    I was actually planning on [laughs] doing something like that, but-

55. 8:42

    Fair

56. 8:42

    ... as you know, we didn't have time to do that, so-

57. 8:45

    Yeah, I know

58. 8:45

    ... I would probably-

59. 8:45

    You're busy, sorry

60. 8:46

    ... yeah. I might have, I might take a crack at that if ... Yeah.

61. 8:48

    And also, is there gonna be a Q&A? I don't wanna ask any more questions.

62. 8:50

    Oh, yes. No, no, there i- So you can log questions on Q&A. There is, there will be Q&A.

63. 8:54

    Oh, you can call

64. 8:54

    So we ... No, no. Yeah. No, you, you, you ... No, you can, like, anyone can, like, raise their hands and, like, ask a question, like, I don't ...

65. 9:00

    I'll just repeat the question. Um, and, but there is, like, the slider if you wanna random questions, I will keep monitoring. Um, yeah. And yeah, so, like, oh, yes, the other [REDACTED:generic_id] was, like, another paper called LoRA learns less and forgets less.

66. 9:13

    It shows that, you know, fine-tuning via LoRA does not really work for learning new knowledge and, um, well, it depends. Like, it depends on how you actually read the paper.

67. 9:22

    Like, some components were incorrect. They didn't actually train on all linear layers. They kind of forgot a few. And they also ... You need to do some sort of, like, of, like, you know, special parameters to make this work, and we will also be talking about that as well.

68. 9:34

    But I was just trying to show you that, you know, we don't just do language models, so we have, like, a whole wealth of knowledge across different industries and stuff.

69. 9:41

    Well, not, not industries, topics. Um, and you can ask me any question that you like.

70. 9:46

    Um, so Unsloth, yes. Um, we launched just last December. So I launched this with my brother. Um, it's a bit outdated, the stats [laughs], but anyways, um, I think we have 11.9 something K or something.

71. 9:55

    I don't even know now. But anyways, um, we launched this last December. It generally makes fine-tuning of, um, language models like Llama, Mistral, Gemma faster, to [REDACTED:generic_id] times faster, um, generally speaking, and with, like, sep- 80% less memory usage now.

72. 10:09

    We have, like, some new methodologies which reduce memory even further. And the trick is there is no degradation in accuracy. So like, you know, we don't do any, like, approximations.

73. 10:17

    That's the whole purpose of doing the optimizations, is we don't wanna lose any accuracy. Um, and so we do, like, Triton kernels. So this is, like, from OpenAI. They, um ...

74. 10:26

    It's a language to do, like, CUDA programming. Um, essentially it's like a in- intermediary between the CUDA code and Python itself. And we'll be showing some Triton code. Um, I don't know if we have time for, like, programming Triton, but that'll be another topic.

75. 10:43

    Um, and yeah, so, like, the whole purpose of Unsloth is to make everyone be able to fine-tune their language models with very bad GPUs, right? So, like, Tesla T4s, the free Goo- Does anyone ...

76. 10:54

    Like, people do know that Google Colab has free Tesla T4s, right? Yes? Yes. Right. 65 teraflops, right? It's actually not that bad, um, if you use it properly. Um, just a reminder, there is a common misconception that the P100s on Kaggle is faster.

77. 11:09

    That's actually not correct. Um, P100s I think are five times slower than Tesla T4s. Um, so although it's actually more expensive as a GPU, I think, I think, um, but it's actually slower for ...

78. 11:20

    Um, so please do not select the P100s on Kaggle, right? 'Cause so Kaggle has [REDACTED:password] hours for free per week GPUs. Um, and you get [REDACTED:generic_id] Tesla T4s, so that's 130 teraflops per week, um, [REDACTED:password] hours, and that is actually very powerful.

79. 11:35

    I think that's the same as RTX 3070, although I can't ... Yeah, I can't, I can't remember exactly. But, um, yeah. So that ... So Kaggle has [REDACTED:password] hours for free per week.

80. 11:45

    Google Colab, it depends on how much you use. Normally you get four hours per day, I think. Um, I guess the Pro is not that bad, it's, like, $10 per month.

81. 11:53

    You can actually get, like ... Yeah, it's pretty good. Um, yeah, so probably get Pro. Um, I do not suggest ... I mean, you could use, like, RunPod and Lambda Labs and stuff like that.

82. 12:02

    I, I guess that's another option. But, uh, we do actually share a pricing comparison. So what you need to, like, be careful when you use GPUs, there is a big issue, is like, oh look, this is the most ...

83. 12:12

    I want to use a H100 Um, did you actually check how much FLOPS the H100 provides? Um, be careful of NVIDIA's marketing, it's times [REDACTED:generic_id] because it has sparsity.

84. 12:23

    Um, so just be careful of that. And also, you have to be careful of the FLOPS when it's like float eight or float 16, so just be careful of those.

85. 12:30

    Um, I do have a pricing comparison where we, like, normalize by, like, the FLOPS, um, with no sparsity, and we'd, like, look through, like, Lambda Labs, RunPod, Google Colab, AWS, um, Google Cloud, and I think RunPod is m- mostly pretty good.

86. 12:44

    Uh, yes, question.

87. 12:46

    Why is it times [REDACTED:generic_id]?

88. 12:48

    Oh, the sparsity?

89. 12:49

    Yeah. Why?

90. 12:49

    So in... Oh, okay, so the question was, why is times [REDACTED:generic_id] for FLOPS for the sparsity feature on NVIDIA, um, GPUs? So sparsity, the sparsity feature, what it does is you take 50% of the weights and make them go to [REDACTED:generic_id], and NVIDIA essentially allows you to train this [REDACTED:generic_id] times faster by not doing matrix multiplications on

91. 13:08

    the zeros. Right? So you're like, [REDACTED:generic_id] times [REDACTED:generic_id] is just [REDACTED:generic_id], so, like, you essentially don't fire the transistors, and essentially this makes it [REDACTED:generic_id] times faster. Um, well, that's actually not...

92. 13:16

    That's just a high-level overview, but, like, essentially you, like, compress the matrix into this special format, and then this NVIDIA special format allows you to do matrix multiplications [REDACTED:generic_id] times faster.

93. 13:25

    Um, yeah, so it's sparsity. Does that-

94. 13:30

    Is it only on the H100?

95. 13:31

    So it's on H100. It's on, it's on A100s as well. So your RTX 3060, your RTX [REDACTED:password] series has that feature from RTX [REDACTED:password], the [REDACTED:password] series. I think it has.

96. 13:40

    Yes, I think it does. Um-

97. 13:42

    And we can, we can enable it or no?

98. 13:45

    So if you want to enable it, the biggest issue is, is that the language models, most companies do not train their language models with sparsity enabled. If you set this, like, weights go to [REDACTED:generic_id], um, you will actually ruin the behavior of the model.

99. 13:58

    So there are, like, papers which show that you can actually do... Like, turn on the feature, and then you can do fine-tuning to make it work. So there are actually papers which do that.

100. 14:07

     Um, so in theory, you could enable this. Um, but, you know, it de- it depends on, like, what models are released from the large companies. Um, yeah. I'm assuming...

101. 14:16

     I, I think I do know if... So Facebook is, like, they did implement sparsity in their PyTorch and, um, xFormers library, so I'm assuming they might be focused on sparsity, 'cause you get times [REDACTED:generic_id] faster.

102. 14:28

     Um, and if you know, like, OpenAI, like, you know, like, they keep saying, "Oh, it's [REDACTED:generic_id] times faster. Hmm, I wonder why. Why is it [REDACTED:generic_id] times faster? Why is it the [REDACTED:generic_id] times faster?"

103. 14:36

     Right? So, like, could it be sparsity? Could it be float eight? Right. Float eight is generally [REDACTED:generic_id] times fast. Okay, not, not exactly, but, like, approximately [REDACTED:generic_id] times faster.

104. 14:44

     So, you know, all these, like, things when you hear [REDACTED:generic_id] times faster, where does it come from? Could it be these things? Although we don't know, but, like, we're just guessing.

105. 14:51

     Um, yeah. Any other questions? Okay. Just remember, you can raise your hand or you can just sli- Wait, are there any... I'm assuming there is no Slido questions yet.

106. 15:01

     Um, yeah, okay. Yeah. It's just raise your hand. Um, so we, so for Unsloth, we do benchmarking against Hugging Face plus Flash Attention too, um, and we just show our benchmark.

107. 15:11

     This is kind of old already. Uh, the memory reduction's much more now. Um, so... And we do a blog, uh, we did a blog post with them, so thanks to Hugging Face for the collaboration.

108. 15:19

     Um, and essentially all you need to do is, you know, from inslo- from Unsloth import fast language model, and we try to make it as easy as possible for people to fine-tune a language model.

109. 15:30

     Um, and yes, we, we talk about Unsloth a bit later. Um, oh, there is a question. Is it a myth or solid hypothesis that linear versus cosine then short [REDACTED:generic_id] to [REDACTED:generic_id] epochs versus three to five epochs is the highly, is, or is highly generalized?

110. 15:45

     Um, I think it depends. Um, so, like, for training methodologies-

111. 15:49

     Sorry. So is it a myth-

112. 15:51

     Oh

113. 15:51

     ... or so- solid hypothesis that linear versus cosine, then of course short epochs versus long epochs, is the highly generalized best way to train as any standard base model?

114. 16:04

     I think it depends. So there are, like, some research papers which show that cosine or linear schedules... I mean, it depends. To tell the truth, I think it's a toss of a coin.

115. 16:13

     I don't think so is actually that important for the l- learning rate scheduler. I think it more depends on, like, the dataset, the number of parameters. Um, there are, like, research papers which show that if you just simply change from tied weights, from untied weights to tied weights, you can get, like, better accuracy for smaller models.

116. 16:29

     So I think it's... So the learning rate schedule is not that important. You might get accuracy plus 0.1%. Just train for more data. There we go. Right? Get more data.

117. 16:37

     Oh, wait, I pressed back. Just train for more data, and I'm assum- I'm assuming it will be similar. Um, but to tell the truth, I think it's best to, like, do small experiments and then, like, test which schedule is the best, but I don't think so it's that, that important.

118. 16:48

     Um, I think for the, the number of epochs, that's actually important. Um, these, like, big companies, um, to tell the truth, I'm not sure what Llama, like, 15 trillion tokens, is it actually 15 trillion tokens, or is it, like, 5 trillion tokens times three epochs?

119. 17:02

     I do not know, right? These are, these questions are very important. If it's 5 trillion tokens times three, that's actually very different from actually 15 trillion tokens in total, right?

120. 17:10

     That's actually very, very different. But in general speaking, like, you know, if you train for more epochs, three is generally the, well, you know, good, like, approximate three epochs.

121. 17:19

     [REDACTED:generic_id] is actually the best for pre-training generally. Um, you shouldn't, like, retrain your data, like, multiple times. Um, but yeah. Um, so did you have a follow-up question or no?

122. 17:28

     Well-

123. 17:29

     Yeah

124. 17:29

     ... um, so basically, um, learning rate is [REDACTED:generic_id], was [REDACTED:generic_id] of the big issues you fixed with the Gemma implementation.

125. 17:38

     Oh, yes.

126. 17:39

     So that's why I kind of... That's where, um, my pitfall was when I was training my 2B for, um, Gemma. And, um, so I actually trained it pre your fix-

127. 17:51

     Hmm

128. 17:52

     ... and somehow it turned out the benchmark and, and after your fixes better than... I don't know what happened, but it, it now is, like, [REDACTED:generic_id] of the highest ranking benchmark 2, 2Bs.

129. 18:04

     Oh, okay.

130. 18:05

     So I don't know-

131. 18:06

     I... Yeah

132. 18:06

     ... like, what, what... Do you have any theories about what could have happened? I trained it on the transformer RoKE converter-

133. 18:12

     Hmm

134. 18:12

     ... and then subsequent using Axolotl and- A, um, the, a very hard... We brute forced a learning rate-

135. 18:20

     Mm

136. 18:21

     ... basically. But it turned out surprisingly well and, and, um, we also u- we didn't use Unsloth, but we used, like, uh, 8 GPUs, so, um-

137. 18:31

     So this is after the fixes that we did? Like it does-

138. 18:33

     No, before. That's-

139. 18:33

     Before. So before does better?

140. 18:35

     No. Well, no, before it was, it was usable.

141. 18:38

     Mm-hmm.

142. 18:38

     Everybody else's was u- was unusable, right?

143. 18:40

     Yeah.

144. 18:40

     But it was usable-

145. 18:41

     Mm

146. 18:41

     ... which was a surprise to me.

147. 18:43

     Mm.

148. 18:43

     Because it, because everybody else's was unusable. But then after your fixes, we are now the top... Well, my company has, on the OpenLLM leaderboard, has the highest-

149. 18:54

     So y- so you didn't even retrain. You like, you just-

150. 18:56

     No.

151. 18:57

     Oh.

152. 18:57

     So after the fixes, somehow it got better.

153. 18:58

     It made it better? Okay.

154. 18:59

     Do you have any theories on that?

155. 19:01

     To be honest, I do not know. I think, like, because the fixes that we did for Gemma are, like, multi-pronged. Like, it's not, like, [REDACTED:generic_id] fix, it's, like-

156. 19:07

     Okay. I see

157. 19:07

     ... nine or something. So I don't know which fix caused the change, [laughs] or they could be, like, all of them maybe. I don't know. Um-

158. 19:13

     I've, it, it's, it's kind of like, um, just for me when I, when I, when I did the training, right, and it turned out good-

159. 19:20

     Mm

160. 19:21

     ... and then I heard from, like, every, all of my friends, "Oh, I can't do this. I can't do this."

161. 19:24

     Mm.

162. 19:25

     It's like, um, it, it just shocked me, I guess.

163. 19:27

     Okay. Yes, that is quite shocking. If, like, you didn't... Yeah, we changed the code. Like, we, we kind of, like, fixed all the issues, and then you don't need to retrain it and it does better?

164. 19:36

     Okay, that's, okay, that's a very interesting phenomenon. I do not know. Un-

165. 19:40

     I sent you the code for the-

166. 19:41

     Oh, yeah. Okay

167. 19:41

     ... it's actually open source on my, on the, on-

168. 19:44

     Okay. Yeah, great. Um, yes. To be honest, like, language models, I mean, these are all active areas of research. Please someone, will you do a research on that?

169. 19:52

     I, I'm a method-

170. 19:52

     Yeah

171. 19:52

     ... to seek interpretability in there.

172. 19:53

     Okay. Yeah

173. 19:54

     ... so I like these, like, super low-level stuff.

174. 19:56

     Yes. I cannot say anything other than I just read the code and fixed the bugs. I do not know. Uh, [laughs] so-

175. 20:02

     Sorry. Sorry.

176. 20:03

     Yeah.

177. 20:04

     Paul will know when you get there.

178. 20:04

     Yeah, yeah. Don't worry. Um, okay. We also, like, do long context fine-tuning. So, like, we show that if you use a new methodology which does gradient checkpointing and offload it to system RAM, you can randomly increase your context size by four.

179. 20:19

     And your... And the weird part is if you offload correctly to system RAM from the GPU, weirdly the time of execution is just slower by 1 to 2%. Right?

180. 20:31

     So, like, this is very weird. It's like if you can do non-blocking calls and offload the GPU, like, memory into system RAM, if you do it correctly, it's not slower.

181. 20:42

     Um, some implementations unfortunately offload incorrectly. Um, I don't want to name anyone, but they offload incorrectly. Sometimes they offload to disk. I don't know who came up with the idea of offloading to disk.

182. 20:53

     But anyways, please try to offload to memory first and then disk, right? Disk is extremely slow. Um, and if you can offload to memory system RAM, you can actually get away with a lot of memory usage.

183. 21:04

     Um, okay. So I should have put this at the first slide, but anyways. So today we'll be having three approximate topics. Um, and these, I wanted to make them into, like, three different separate topics, but I guess I just mixed them together.

184. 21:17

     Whatever. Um, so you'll be learning about low-level technicals of language models, for example, backpropagation. Um, why is, why is transformers not O of N square... Oh, sorry, O of N cubed for training, and rather O of N squared.

185. 21:32

     And there is a lot of maths, but I will try my best to reduce... I think I already tried my best to reduce the maths, but there is still some maths.

186. 21:37

     So please handle the maths. Um, I will try my best to explain as simply as possible. Um, that's the whole goal of the workshop, so not that bad maths.

187. 21:47

     You will actually understand the formulas very well. Um, just a reminder, I've kind of nearly failed my maths in university, so do not worry, do not be scared. Um, it's very fine.

188. 21:57

     Um, we are talking about Unsloth fa- fast fine-tuning, the best tips and tricks for fine-tuning, how do we write the fast kernels for fine-tuning, um, you know, how do we actually make it [REDACTED:generic_id] times faster, use 70% less memory, like how, uh, with no accuracy degradation.

189. 22:11

     Um, and we'll be talking, like, some, you know, Triton, OpenAI's Triton language and stuff like that. Um, and we'll be doing finding and fixing bugs. Um, so this will be a constant phenomenon and theme.

190. 22:21

     How do we find and fix bugs in Llama, Mistral, Gemma, Phi? Um, we'll be talking about Mixture of Experts as well a bit. Oh, wait, maybe not, but it depends on time.

191. 22:29

     Um, and we'll be doing lots of bug hunting, bug fixing, and more. And everyone here will be a fantastic bug hunter and bug fixer. And we can, like, essentially open source our effort to fix open source models to everyone here.

192. 22:40

     Um, oh, yes, and we also have stickers. Um, yes. I, I don't know where they are, but, like... Oh, yes. Yeah, my brother has some stickers. Um, and we bought a few of these stickers which look pretty cute, right?

193. 22:52

     So, like, you can... Oh, wait, my laptop has some, right? I put them on my laptop. Um, and [laughs] they're pretty cute. I r- really like them. My bro- Um, so my brother has them.

194. 23:01

     We'll be handing them out, um, yeah, as well at the end. Um,

195. 23:07

     okay, so let us start. Um, so the transformer, right? So, like, what is the transformer? I'm assuming everyone knows what the transformer is. Um, does anyone not know what the transformer is?

196. 23:20

     Yes or no? Like, you can simply-

197. 23:21

     I, I, I more or less.

198. 23:23

     Okay. Yes. Okay. So the transformer is just an architecture that is behind all language models. Um, so, like, GPT-4, GPT-3, you know, like, Llama, Mistral, Gemma, all these open source models, what are they?

199. 23:36

     Like, what's the architecture behind them? And all of them are rely on the ar- like, the transformer. And the transformer is essentially a architecture which seems to be very good for sequence modeling.

200. 23:48

     Um, so it's not just for languages, it can be for any sequence modeling, right? So, like, if you know, like, you know, Sora is a transformer. Well, not just a transformer, probably plus diffusion, but, like, it's generally transformer.

201. 23:58

     Um, and there's other different types of models which doesn't have to be language modeling, okay? It's just sequence modeling. And I'll probably show some pictures later. Um, I probably should have explained a bit better, but, like, just, just, just assume that transformers are the method behind all language models, okay?

202. 24:15

     GPT-4, GPT-3, GPT-5. Okay, I don't know if GP- Okay, who knows what GPT-5 is? But, like, I'm assuming it's a transformer. [laughs] Um, the transformers just do seem to be very good at learning new knowledge, injecting knowledge into the model.

203. 24:29

     It seems to be very, very good at changing the weights to fit the training data. Um- Which is very interesting. Um, and the GPT-2 architecture was actually very popular for the f- like, you know, most a decoder style transformer.

204. 24:43

     That was very, very popular. It's still used to this day. Um, and it kind of got reincarnated by adding extra components to it, and this new architecture is called Transformer Plus Plus.

205. 24:54

     Um, I don't know if people have heard of this, but Transformer Plus Plus is the GPT-2 transformer architecture, plus RoPE embeddings, plus SwiGLU, plus RMS Layer Norm, and with no bias.

206. 25:04

     Um, and I think it's untied weights, although I'm not sure if GPT ... I can't remember that exactly, but I think it's plus untied weights. Um, and Transformer Plus Plus is the architecture which most people think is the best, you know, transformer architecture for now.

207. 25:19

     Um, yes, for now. Um, there are probably, like, some other tweaks and little small things that Transformers still can do, but, like, in general, this would be the considered the best architecture.

208. 25:29

     Um, and how does the architecture look like? It is just a list of maths equations. Um, so I just wrote down the entire transformer architecture, um, well, this is Llama 2, right?

209. 25:40

     So Llama's transformer architecture in [REDACTED:generic_id] slide. Um, and all you need to do is get some inputs, like some sort of, like, inputs, do some Layer Norm, do some RoPE embeddings, do some attention, plus some residual, do some Layer Norm, SwiGLU, whatever, residual, Layer Norm, and you get logits.

210. 25:59

     Um, and you essentially repeat this middle s- section L times or many times. Um, and that is the transformer architecture. Um, okay, then maybe the maths equations, I'm not sure ...

211. 26:09

     Uh, does the maths equations scare anyone or ... I'll be explaining each [REDACTED:generic_id]. Okay. So, like, hopefully I try to make the maths equations as, like, reasonable as possible.

212. 26:18

     Um, in theory, if you write this down on, in PyTorch, um ...

213. 26:23

     Oh, yes, if you write this down in, in PyTorch, you actually have a working implementation of, of a transformer architecture. Um, and yeah, so, like, we'll be talking about each component separately as well.

214. 26:35

     Um, yeah. Is anyone scared for the maths? No. Yes. No. No. Okay, very good. Okay, let me just check questions. Does anyone have any questions?

215. 26:44

     Okay. Okay, so ... Oh, did you have a question? Okay.

216. 26:49

     Well, um, so from my understanding from the layer level for transformers, um, so it's almost comparable to cosmic Plinko in a way. Is that ... I mean, of course, with the hard math behind it, but-

217. 27:04

     Cosmic, sorry, what?

218. 27:05

     Cosmic Plinko, like the drop the ... You know, because there's 80 layers, right?

219. 27:10

     Mm-hmm.

220. 27:10

     In a grid, right?

221. 27:12

     So, like, low energy something? Sorry, I'm, I'm actually not familiar with that. Do I ... You'll have to explain to me.

222. 27:18

     Hmm?

223. 27:19

     Try ... What is this, cosmic what? Sorry.

224. 27:21

     Cosmic Plinko. Uh-

225. 27:22

     Classic Cosmic Planko? Do I know that? I, I built that spot [laughs]. Sorry, you have to, like, explain to me what that is.

226. 27:28

     Um, um-

227. 27:28

     Should I search that up?

228. 27:29

     Gotch- not gotcha. Um, like, uh, it's, it's the arcade game with a bunch of pegs and you drop a ball and it bounces off pegs.

229. 27:35

     Oh, that. Oh, yes.

230. 27:38

     It, it-

231. 27:38

     It's a Windows XP

232. 27:39

     ... the output layer, uh, output, you know, on the bottom right-

233. 27:42

     Yes

234. 27:42

     ... you should visualize it, right?

235. 27:44

     Yes.

236. 27:44

     Like, for, if you take this and visualize it-

237. 27:46

     Yes

238. 27:47

     ... and then, and then visualize the math ... For example, let's take Llama 3, uh, 88B, right?

239. 27:52

     Mm-hmm.

240. 27:53

     It's 32 layers, right?

241. 27:54

     Mm-hmm. Yes.

242. 27:55

     And then layer [REDACTED:generic_id], or I guess there ... It doesn't matter. There's layer [REDACTED:generic_id] and then layer 32, which out- is the output layer, right?

243. 28:02

     Yes.

244. 28:03

     Now, when you drop a, uh, prompt into this cosmic machine-

245. 28:08

     Yes

246. 28:08

     ... which it's not actually cosmic, we just don't understand-

247. 28:12

     Okay, yes. It's not cosmic. Yes. It's just maths. Yeah.

248. 28:14

     But so far it's cosmic to us.

249. 28:16

     Yes.

250. 28:17

     I don't think we understand ... Personally, I don't think we understand 50% of it yet. Like, we're, we're-

251. 28:22

     I know. I think we've learned a lot, but okay, yes. [laughs] Okay. This is maths.

252. 28:25

     I, I ... So I g-

253. 28:25

     Yes.

254. 28:26

     You're the expert, not me. I'm, you know, so-

255. 28:28

     Okay

256. 28:28

     ... but, um-

257. 28:29

     So you're just trying to say an analogy, like it's kind of like the game?

258. 28:32

     Yeah. So, so if people can visualize the math versus, like-

259. 28:37

     I, I think-

260. 28:38

     Is that a fair comparison?

261. 28:38

     So we ... Oh, yeah, I will talk about that in the slides, but, like, I think it's more like,

262. 28:42

     hmm, an analogy. I think it's more like you're going through, like, a ma- like ... Okay, it's not a maze. I would say it's more like you have, like, a ...

263. 28:50

     Every single layer has, like, someone trying to make you change clothes, and then each layer there's, like, a fashion designer trying to get you to wear different clothes, and each layer, the fashion des- designer doesn't like the previous fashion designer's choices, and they will change your clothes.

264. 29:07

     Something like that. I think that's more like a transformer. It's like each fashion designer has, like, views of their own.

265. 29:13

     So it's actually quite similar, kind of, because-

266. 29:15

     Okay. Yeah. I guess you could say

267. 29:16

     ... Plinko does have, like, kind of, like, pegs on it. So, like-

268. 29:18

     Oh, yes, yes, yes. Wait, it is in Windows XP, right? Is it that? I'm, I'm maybe confused. Windows XP, like the game? I think I played it before.

269. 29:25

     It does-

270. 29:25

     Oh, okay. Anyways. Okay. Yeah, sorry. Yeah, question.

271. 29:28

     In the notation, is there a way to tell what's a matrix or a vector or

272. 29:33

     Oh. Um, so when I put subscript I, it generally is a ve- Well, technically everything's a matrix, but if you see any summation signs, um, with subs- subscript I, then it gen- generally means row wise or column wise.

273. 29:48

     Um, the W, like, if you see small, if you small, small s- like a small w, that generally means vector. Um, but in general, everything that's capital is a matrix.

274. 29:58

     Um, and why is it a matrix? Because it's just faster. Um, I mean, in theory, you could convert this all to vectors, but for speed purposes, this should be matrices.

275. 30:10

     Um, yeah. Any other questions? Okay. Next. Um, so why did I put this, "Hello, my name is Daniel. Hi, my brother's name is Michael. I hope everyone will have tons of fun.

276. 30:23

     AI Engineer World's Fair is the best." Um, why did I put this? Well, does anyone noticed any

277. 30:31

     similarities between these sentences, um, or differences Sorry?

278. 30:39

     Exclamation mark at the end.

279. 30:40

     Ex- okay, yes. Okay, yes. Okay, except for the first sentence. Okay, anything else?

280. 30:46

     Just say random stuff.

281. 30:49

     I mean, uh, hello and hi are the same thing, just kind of like-

282. 30:52

     Interesting, yes. Okay, hello and hi are the same thing, but kind of different words. Okay, yes. Yes?

283. 30:56

     Semantic embeddings. Basically, this is a example of semantic embeddings.

284. 31:01

     Okay, okay. Somewhat, okay.

285. 31:03

     Verbs are all similar. Is, is, will, have are all to be verbs.

286. 31:08

     Okay, yes. Okay. Okay, well, to, to, to tell the truth, I didn't have any intention [laughs]. The intention... Yes, go on.

287. 31:14

     I mean, it's the king plus queen-

288. 31:17

     Yes, yes, yeah, for word2vec. Yeah, yeah.

289. 31:19

     Yeah.

290. 31:19

     I know what you're talking about, yeah.

291. 31:20

     So, like, you know, it's king minus man plus woman equals queen. That's the... I think that's what you're trying to show a little. May- I mean, maybe you don't have that intention, but this kind of, it shows that a little bit.

292. 31:33

     Okay. Well, in theory, I guess you could have just seen the next slide. There are just five [laughs]. If you simply just look, um, the first [REDACTED:generic_id], if you consider punctuation as combined with the word, right, like hello comma, treat that as [REDACTED:generic_id] separate s- component.

293. 31:50

     And if you do this, ignore all spaces. The first [REDACTED:generic_id] has just five components, right? What do you think the second [REDACTED:generic_id]... Okay, I guess you can already have the slides, but what do you think the second [REDACTED:generic_id] has then, right?

294. 32:01

     Anyone? I already wrote all the answers there.

295. 32:06

     You did.

296. 32:06

     Okay. Yes, I wrote all the answers, right. Six, eight, seven, right. [REDACTED:generic_id], [REDACTED:generic_id], three, four, five, six, seven, eight, right? [REDACTED:generic_id], [REDACTED:generic_id], three, four, five, six, seven, right. So, like, if you do this, um, just assume all punctuation from now on is combined with the word and ignore all spaces, um, and this, we just invented a tokenizer,

297. 32:26

     um, right. So this is a very general, a random tokenizer we just invented. Um, and each [REDACTED:generic_id] is- has an... Essentially, the reason why we wanna do this is because computers doesn't, they don't understand words, they only understand numbers, right?

298. 32:41

     So, like, you have to essentially assign each of these tokens as a number, like an ID, right? So hello is ID [REDACTED:generic_id], right? H- hello comma actually is ID [REDACTED:generic_id].

299. 32:50

     My is ID [REDACTED:generic_id]. Name is ID [REDACTED:generic_id], and so on, right? So, like, you have, you have to assign an ID to e- each of these components. If you don't do that, then a computer doesn't know what you're actually doing, right?

300. 33:01

     Computers only know numbers. So we just invented a tokenizer. Um, I would not suggest you to use this tokenizer, but in general, it's actually not that bad, um, 'cause can anyone, uh, see any issues with this new tokenizer we just created?

301. 33:16

     Um, what are some issues? Yes.

302. 33:19

     If you were, um, if you include the punctuation, the tokens for Michael with and without punctuation would be different tokens.

303. 33:25

     Yes, very good. So what do you think we should do? Okay, so the point was we included the punctuation for the words, and that is not helpful, right? So, like, Michael, explanation mark, or Michael, not explanation mark.

304. 33:36

     So what would you suggest to fix this?

305. 33:39

     Um, I would assign tokens to the punctuation.

306. 33:41

     Interesting. So hello and then hello will be [REDACTED:generic_id] token, and then comma itself will be a-

307. 33:47

     Yeah.

308. 33:48

     Okay, interesting. Anyone have any other suggestions? How about... Yes.

309. 33:52

     So the point of the tokenizer is to reduce the vocabulary, but here you haven't reduced the vocabulary, right? Because every new word will be a new token essentially.

310. 33:59

     Yes, very good. Exactly. So do you have any suggestions for how to improve?

311. 34:05

     Um, you could stem the word. That would reduce the vocabulary.

312. 34:07

     Oh, yes, very good. Yes. I haven't heard that in a long time. Very good. Yes, old natural language stuff. Um, so the idea was to stem the word. So essentially you can remove like, you know, for example, like skipping can become skip, right?

313. 34:20

     So, like, skipped, skipping, skip, like, you know, they're all kind of the same... Well, I wouldn't say the same thing, but, like, in theory, they're the same thing. Um, any other suggestions?

314. 34:28

     My and My with a capital M are treated differently, but they mean really the same thing. So can you just lowercase them all first?

315. 34:36

     Very good idea, right? If you lowercase them all, then you can reduce a lot of issues. Um, is that a good idea? Capital My and small my, what do you think is the difference?

316. 34:47

     If you do capital My generally means it's the start of the sentence. If you do my with no lowercase, then it means it's the middle of the sentence. So good idea, though.

317. 34:55

     Um, actually, I think, I think, um, so but is an old... Okay, people still use but. Um, I think there is a lowercase version of but, so they essentially lowercase everything.

318. 35:06

     Um, and I think it does okay for, like, semantics and stuff, but it doesn't really do well for dec- decoder type style. So don't lowercase but, good idea. Any other suggestions?

319. 35:16

     Uh, yeah.

320. 35:17

     Uh, you build a vocabulary by starting with just the individual tokens instead of-

321. 35:23

     Mm.

322. 35:24

     S- size for how large your vocabulary would be.

323. 35:27

     Yes.

324. 35:27

     Then you start building word pieces from the most frequently observed.

325. 35:32

     Oh, you just said the name of the algorithm. It's called WordPiece or BPE tokenization. [laughs]

326. 35:38

     Yeah.

327. 35:38

     Um, yes, exactly. So that's actually the correct... Well, I wouldn't say it's the correct way. I actually don't like that approach. But yes, it's the most recognizable and most industry standard approach is to do what you suggested, which is to start off with, like, small little individual characters and then combine them together based on some sort of

328. 35:54

     statistic, right? You, you shouldn't... Like, you know, hello. Like, maybe hello is a word that we have to select, right? So, like, but, uh, but, like, H-E... Like, H-E-L-L.

329. 36:06

     Okay, hell, okay, that might be a popular word as well in, like, the dictionary, but anyways. Or he, right? So he, hell, hello, um, each of these might have assigned different tokens, but they might not be, right?

330. 36:16

     So, like, it depends. Um, and the correct industry standard is to use these methodologies to build up the component, um, which we won't be discussing today, but, like, that's for later research.

331. 36:27

     Um, yes. Okay. So let us just look at [REDACTED:generic_id] sentence, right? The first [REDACTED:generic_id] is, "Hello, my name is Daniel." Assuming our tokenization's useful... Okay, let's just assume, uh, the tokenization which we created is helpful.

332. 36:44

     Okay, so remember, it is just to- Put all the punctuation together, um, ignore spaces, and don't do lowercase or whatever, just, just, you know, tokenize it as it, as it is.

333. 36:55

     It's not very useful, but who cares? We'll just s- say this is a good tokenizer. Um,

334. 37:01

     the question now is, if I select the first token, hello, right? Let's assume... I don't know if... Okay, the color's not very good, but, like, my name is Daniel is grayed out.

335. 37:09

     I, I think... I'm not sure if you can see that grayed out. Pretend someone types in hello, comma, right? A language model should predict what's the next word, right?

336. 37:18

     So how does it know to predict mine? Right, you know, when you type ChatGPT, it, like, it goes from left to right. Like, you type something, you know, you type some sort of instruction, and then ChatGPT will, like, print the words from left to right, right?

337. 37:29

     So, like, why is it printing words? Can anyone tell me why is ChatGPT printing words from left to right? Why is it not doing right to left? Or why does it not just spit out everything in [REDACTED:generic_id] go?

338. 37:39

     Yes.

339. 37:40

     Because it's, uh, basically a sequence modeling, and it's just taking, for this case, hello, and then predicting what's most likely the next word depending on

340. 37:50

     Exactly. Correct. So because it's a transformer type, decoder t- type architecture, it is predicting the next word based on the previous words. So very good. Um, and so the point is, is we only can have the language model see previous words, not future words, right?

341. 38:10

     If you accidentally put the future data in, oh, you know, your accuracy might go to 100% if you use future data, so please do not do that. This is actually a very common mistake.

342. 38:20

     Like, I'm not saying this jokingly, it's a very big issue in research. So please read the paper before you actually... Like, if you read research papers, please see how they do the methodology.

343. 38:30

     Um, always read... Did they put the fu- Always ask the question, did they put future data in when they did the training or doing the research paper? Okay, this is a pervasive problem.

344. 38:39

     I'm not joking. Um, you can see, like, weird accuracies. Like, if you see papers which have, like, 98% accuracy, question, question, question, question, question. Okay? Or 100% accuracy. How is that even possible?

345. 38:50

     How can something be 100% accurate? Question, right? And so the most likely scenario is they use future data. Yes.

346. 38:56

     Do you find the, the paper itself will actually catch this? Or I, I feel like I often find that, like, the logic in the paper makes a lot of sense, but then I'm really questioning, like, how likely their code actually implemented what they think they are doing.

347. 39:11

     Right? Like, do you find-

348. 39:12

     Hmm

349. 39:12

     ... more like the paper actually gives you that issue or that the code has some bug in it?

350. 39:17

     That is a very good question. So the, the question was, like,

351. 39:21

     is it the paper, the research paper, like the methodology itself using future data, or is it the code implementation the problem? Now, that's actually a very good question because it depends.

352. 39:32

     So I think if the researchers... Like, I think normally in research papers, [REDACTED:generic_id] person does the coding, some, some other people do, like, secondary coding, and then the main researcher makes the idea.

353. 39:41

     They don't actually do the coding. So the, the, the coder might have misinterpreted the methodology. To be honest, I don't think that that's the case. I think it's actually the methodology that's the problem.

354. 39:53

     I think if the researcher, like the main researcher, if they find out that their m- method has 90%, 98% accuracy, like, why didn't they question the results? Like, I would...

355. 40:02

     Like, wouldn't that be very sketchy? Like 98% accuracy, 100% accuracy. I'm not joking, this is actually a very serious problem. So if you read enough research papers, you'll see this problem is always the same problem.

356. 40:13

     It's always using future data. Um, so yeah. I think the... Yeah. I think, like, if you don't... Yeah. Okay. I won't comment on any papers. Anyways, uh [laughs] yes. Okay.

357. 40:23

     Does that kind of answer your question? So it depends. Um, I think in general, I would say it's the, it is the researcher's, the main res- the lead researcher's responsibility to correct these mistakes.

358. 40:33

     Um, then you shouldn't be the re- lead researcher. So, like, that's my take on that. Um, I think the coder, I, I mean, the programmer might have some issues, but I don't know.

359. 40:43

     I, I think I, I blame the lead researcher. Um, any other points?

360. 40:47

     It seems like you're saying the main red flag is, like, a huge leap in performance.

361. 40:51

     Yes. If you see huge leap in performance, question, question, question. It's probably future data. Actually, I wouldn't say it's probably. It's like 50% sure it's future data. Um, yes.

362. 41:01

     Question.

363. 41:03

     Um, do you think, uh, you know, in methodology like they do like a train validation test split, uh, you know, what if, uh, the way it is sort of like redacted, you know, we shuffle the data-

364. 41:13

     Oh, yes

365. 41:14

     ... and the tokens, like, whatever we are calling future data, what if, like, the tokens, name, entities and stuff, like, they, they didn't actually shuffle that right and that was all in the training data and the test data or the validation split, whatever that, um, that, that just essentially have tokens that they've already seen, so it's able

366. 41:31

     to do a good job and get 98%?

367. 41:33

     Exactly. So the point was, um, in the test and train split, sometimes, although you might not be using future data, you might, like, accidentally put different components in the tr- like, the training set from the test set, and then depending how you split the data set, you might actually mix the data sets.

368. 41:50

     Is that kind of correct?

369. 41:51

     Yeah. It's like basically are you biasing basically your-

370. 41:54

     Yes. The distribution itself.

371. 41:56

     Yes.

372. 41:56

     Yes. So that is why you should-- you have to be very careful to how you split the data sets, right? For training and testing. You have to use stratification, you have to inspect the data before you do it.

373. 42:05

     You must enable random shuffling, right? There is actually a very, very common question on PyTorch. How do you not enable random fan-- uh, uh, for, for Hugging Face? How do you not enable random shuffling for Hugging Face?

374. 42:17

     Um, they purposely disabled this. Like, think about, like, why. Because, like, people might forget to randomly shuffle. Um, and actually, this is a pervasive problem with Kaggle competitions. People like to, like, train on the tests.

375. 42:29

     Like, they give you, like, 20... Normally speaking, you should have 20 test, 80 train, right? So, like, but people like to, like, the final submission, they use even the test in the submission.

376. 42:38

     So, you know, um, it is a pervasive problem. Um, any other points and questions?

377. 42:45

     Okay. So what is a language model? Given the word hello

378. 42:50

     Can you somehow predict my name is Daniel, right? So like given the word, right, so given that token, can you predict these extra tokens? Um, remember

379. 43:00

     carefully I purposely only rep- went from left to right. Right? So like hello can only predict my, can only predict name, is, and Daniel. Right? It cannot predict... Right?

380. 43:10

     You can't use the word hello to predict previous words. Right? That's cheating. So that is

381. 43:18

     m- sequence modeling. And the point is, when you chain them together, let's assume that you predicted the word my as the next word, right? So now you have [REDACTED:generic_id] pieces of information.

382. 43:28

     The text that you see is hello, comma, my. Using these [REDACTED:generic_id] pieces of information, you want to predict name is Daniel, and you keep doing this, and that is a language model, right?

383. 43:39

     That is essentially a language model. What is it doing is you start from the first word, you predict the words into the future, um, and you keep doing this iteratively, right?

384. 43:49

     And so that is what ChatGPT is doing. Um,

385. 43:53

     does, does that kind of make sense? Um, and remember, the point is never use future data. Okay, I'm, I'm... I know I keep stressing this, but like this is actually a very big issue in machine learning and AI, right?

386. 44:03

     This is actually the biggest issues I find in my, like, opinion, is using future data. There's so many papers which do this. Um, yeah.

387. 44:12

     So the second point is you must tokenize each component into numbers. Remember this? So I'm just going to, you know, cook up some numbers. Hello will be 0.11, minus 0.123, 102.

388. 44:24

     Okay, just made those numbers up. Um, Daniel is 0.11, 123, minus 0.122. Okay, I just randomly made them up. Um, and remember, you must-- each component must have the same number of numbers, right?

389. 44:41

     So like if hello has three, Daniel must also have three.

390. 44:45

     Can someone tell me how many combinations of... If you assign this case that each number must have three, um, you know, three numbers, how many combinations do you think there can be for each token or for each component?

391. 44:59

     How many combinations? What do you think the answer is?

392. 45:05

     So remember, you can choose any single number

393. 45:08

     in the three numbers, right? 0.11, 0.112, 0.111113, whatever number. How many combinations are possible?

394. 45:15

     It depends on the flow of precision.

395. 45:17

     Depe- okay, let's assume it's infinite precision.

396. 45:20

     There will be infinite.

397. 45:21

     Correct. The answer is infinity. You can do as many as you like.

398. 45:26

     Um, but normally speaking, you should use not three... All right, so now the question is why don't you just use [REDACTED:generic_id] number then? Right, hello can be 0.11. If it's already infinity, 0.11.

399. 45:35

     If you use [REDACTED:generic_id] numbers, isn't it also infinity, right? So what's, what's the problem? Well, please don't-- you should use as many numbers as possible. Try your best to use more numbers.

400. 45:42

     It's because the computer, you know, it's not an infinite machine, so please use like more numbers so it can like learn which numbers to like assign it to. Um, and when you start training a language model, these numbers will be randomly initialized, right?

401. 45:57

     So like all these numbers will be randomly initialized. And can someone maybe notice my initialization, what is the issue with this? There are like some-- there is a glaring issue, quite obvious, um, very big problematic issue.

402. 46:09

     If you use the hello and [REDACTED:generic_id] or [REDACTED:generic_id], then you

403. 46:13

     Sorry, sorry. Wait, you, you want?

404. 46:16

     I think there is same number for hello and Daniel.

405. 46:19

     Okay, same number. Okay. Okay, good point. Yes. Did someone say some... What was the other point?

406. 46:23

     They're different in magnitude.

407. 46:25

     Yes, the magnitude is the most important. Right, 123 and 102 are terrible. Um, when you randomly initialize, please do not initialize with random large components. Um, this will destroy your training.

408. 46:39

     That is why your training might have infinities, and sometimes the training loss goes to [REDACTED:generic_id]. Okay, that is not... That does not mean your model learned anything, that just means it's some sort of error in your training data, uh, sorry, your initialization, so be careful of that.

409. 46:54

     Um, don't worry, Hugging Face does this automatically for you, so you don't need to worry. Um, and now each component has a list of numbers that is it, it's associated with.

410. 47:06

     Okay, I just use the same number for now, but it's easier for me to, you know, do the slides. Um, but essentially, hello, comma, my name is Daniel, each of them has numbers associated with them, and this is the thing that you're trying to learn for each of these components.

411. 47:22

     And remember, this can be converted into a table of numbers, right? So like if you replace all the commas with just column, columns, right? These are just tables of numbers.

412. 47:32

     Um, and this table is what you need to train.

413. 47:37

     And again, remember, given the word hello, you want to predict my name is Daniel. Right? So like essentially given that, you know, vector of numbers, can you predict the other vectors of numbers?

414. 47:49

     Um...

415. 47:51

     Just for clarity-

416. 47:52

     Yes.

417. 47:52

     The three dots are?

418. 47:54

     Oh, yeah, you can do as many numbers. So you have to select a option how many numbers you want to select to represent these numbers. So for example, you can select six numbers, or you can select 1,024 numbers or 2,048.

419. 48:07

     It depends on the model creator's choice. Um, yeah.

420. 48:11

     And then every row has to be unique, right?

421. 48:13

     Yes. Every single row must be... Well, it doesn't have to... Okay, not, it might not be the case, there might be not, but it should be, yeah, with high probability it will be unique.

422. 48:20

     Um, yeah. Wait, is it like... Okay. Um, and so when you do training of a language model, um, there's a trick that you use. And remember, we want to predict the next word, right?

423. 48:36

     So hello, you want to predict my, right? So can someone notice any pattern with this? Like why did I do the arrow, and what's the pattern? Does anyone see this?

424. 48:50

     Any special, special like things with this? No?

425. 48:57

     What happens if you take hello, my name is Daniel and just shift it up by [REDACTED:generic_id] place? Is it... Right. If you shift it up by [REDACTED:generic_id] place,

426. 49:05

     hello is now aligned with my. My is now aligned with name, and so on, right? And then there'll be a gap at the very bottom, right? So we simply, we just put EOS, which means end of sentence token.

427. 49:15

     That just means it's the end of the sentence, and we just, you know, put it there because it's a gap, right? So remember, machines d- machines do not like gaps, and you must use all numbers.

428. 49:23

     So that's the reason why we did that. And this is kind of the training mechanism, right? So we essentially, we have that list of, you know, list of words, and we wanna predict the shifted words.

429. 49:40

     And the transformer, all it does is there's a function to predict that. So given hello, can you predict my? Given my, can you predict name, and so on, right?

430. 49:48

     That's the transformer architecture. Um, it's kind of a bit wrong, but, like, something like that, okay? Like, the FX is like this gigantic model that can be like, you know, there's lots of turning knobs in it.

431. 49:57

     Um, okay, let's check if there's any questions. Okay.

432. 50:03

     And the point is, remember, remember the point is we can only use predict the future words, right? So hello can only predict my name is Daniel and so on, right?

433. 50:12

     That's the purple, the purple component. And the blue box is called the attention mechanism. The FFX, I factored it out, and that is called the multi-layer perception or MLP layer, right?

434. 50:23

     So there is actually [REDACTED:generic_id] components in a language model. [REDACTED:generic_id] does the prediction of the next word, and the FFX, which is the MLP, just does this, you know, changing component.

435. 50:34

     Um, it just makes it, like, you know, a bit better than just simple predicting the next word, the attention. Oh, okay, there is a question. Um, oh, I didn't see.

436. 50:45

     Oh, okay. Well, um-

437. 50:49

     Daniel?

438. 50:49

     Yes.

439. 50:49

     When we are, uh, I know we are focusing on the, the most canonical, uh, approach when we are predicting [REDACTED:generic_id] token, but you've got papers now on predicting multiple tokens.

440. 50:59

     Yes, yes. The Facebook.

441. 51:00

     Can you comment on this? I don't want to go into the mechanics, but how, how-

442. 51:04

     So the comment was normal transformers predict only [REDACTED:generic_id] token at a time. How about transformers which predict multiple tokens at a time? Yes, so actually you can predict multiple tokens at a time.

443. 51:15

     It just depends on what is your training objective at the very last layer. You don't have to-- Remember we shifted by [REDACTED:generic_id] place, right? Why don't you shift by [REDACTED:generic_id] places and three places?

444. 51:23

     Then you'll be predicting [REDACTED:generic_id] tokens into the future, three tokens into the future.

445. 51:27

     You'll get multiple... Yeah.

446. 51:28

     Exactly. So before it's just [REDACTED:generic_id] objective, so, like, [REDACTED:generic_id] column of the output. We just add more. Um, and yes, you could do multiple tokens. Um, I've-- Was it the-- It was the Facebook paper, right?

447. 51:38

     I can't remember. Yeah, it was a Facebook paper. Um, I forgot the accuracy though. Um, yeah, but yeah, you could do that. Um, but I guess, like, it-- I, I don't see any...

448. 51:48

     I, I guess it's just good for inference time. Like, you can predict, you know, you can make inference... If you do four tokens in the future, you can predict four times faster.

449. 51:55

     Um-

450. 51:55

     So I think they managed to scale the model down a little bit.

451. 52:01

     Okay. Yeah.

452. 52:02

     Surprise.

453. 52:03

     I think it's mainly for when you do inference, you can do this. If you do [REDACTED:generic_id] tokens in the future, you can have, like, [REDACTED:generic_id] tokens in, in [REDACTED:generic_id] go.

454. 52:09

     Um, I don't know. I, I don't like that approach. I think predicting [REDACTED:generic_id] token is better, um, 'cause you're already forcing the language model to do so much. You're making it even more problematic.

455. 52:18

     So I think predicting [REDACTED:generic_id] to-- Okay, maybe for inference time it could work. Um, yeah. Yes, question.

456. 52:25

     So, um, for the multiple dimensions, is that basically the, for each token, does that correlate to then the dimensions in the layer? And because, you know, tokenizers do convert it into a single number, right?

457. 52:37

     Like, do you actually use a natural tokenizer-

458. 52:40

     So token-- Yeah, tokenizer converts it to a ID. So, like-

459. 52:43

     The-

460. 52:43

     Yeah.

461. 52:43

     That represents then the dimensions, and that's gonna be equal to the hidden layers or it's gonna be-

462. 52:49

     So, so the token- tokenizer, when it has like, when it says like it has thirty-[REDACTED:generic_id] thousand words in the tokenizer, essentially it's an ID from [REDACTED:generic_id] to [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id].

463. 52:58

     Right? So like hello will have ID [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id]. So then what you do is you take [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id], go to the hash table, which has this, which has the, um, this table.

464. 53:08

     Right? I made the table. Right? So hello has an ID, my has an ID, name has an ID, and so on. And you essentially, you hash, you go to that specific row, and then you, that is your, like, sample in your training data.

465. 53:20

     And you do this, like, for the whole thing. So the tokenizer does... You do get a number. It's an integer. Um, and you just have to hash it to the, you know, the embedding matrix, and you'll get like a vector of numbers if that...

466. 53:33

     Does that kind of answer your question?

467. 53:34

     It will correlate to like, say, [REDACTED:generic_id] forty-eight or [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id]. Will that-

468. 53:38

     Oh, you're talking about the embedding di- dimension?

469. 53:40

     Yeah.

470. 53:40

     So the, the... If you see [REDACTED:generic_id] [REDACTED:generic_id] hundred and forty or whatever those numbers, um, that is just the how many, how many, like, columns-

471. 53:47

     Okay.

472. 53:47

     How many numbers you want to represent for each, um, component.

473. 53:51

     Okay. That's the question I had. Thank you so much.

474. 53:53

     Yeah. Okay. Any other questions? Yeah. Yes. Oh-

475. 53:58

     I have [REDACTED:generic_id] on the, I have [REDACTED:generic_id] on the-

476. 54:00

     Oh, okay.

477. 54:01

     So you can answer them la- late, way later.

478. 54:04

     Okay. Okay. Yeah. Yeah. You had a-

479. 54:06

     Yeah. The dimensions, um, just following up on that, I never understood what's the significance of like, like, what is the difference between [REDACTED:generic_id] thousand and forty-eight versus,

480. 54:18

     you know-

481. 54:18

     Four thousand ninety-six?

482. 54:19

     Yeah.

483. 54:20

     I guess it just depends. Like, if you make... So the point-- So remember we said how many combinations can we do? Infinity, because we can do, right? Because floating represe- representation, um, in theory, if you have infinite precision, it can be infinity.

484. 54:33

     But someone mentioned how depending on the precision of your float, it's actually limited precision, right? So, like, the point is you want your model, you want the training objective to actually learn, give it as much freedom as you like, right?

485. 54:44

     So, like, if you're trying to restrict the model when it learns, it might actually not be helpful, and that is why normally people have, like, these large numbers like, you know, six [REDACTED:generic_id] four four embedding dimension or eight thousand [REDACTED:generic_id] hundred and ninety-[REDACTED:generic_id] embedding dimension, right?

486. 54:57

     The more numbers you give it to the model, it just has more freedom to move. Um, so if-

487. 55:04

     In theory, I, I mean, in theory it should have better accuracy. Um, in theory. Everything's in theory. Um, I don't know if there are research papers which show this.

488. 55:12

     I think someone should write a research paper on that. You know, each embedding dimension test, you know, do three trillion tokens ... Okay, that's probably too many, and then see which [REDACTED:generic_id] has a high ...

489. 55:22

     I'm assuming the more you add, the m- the higher accuracy. Yes?

490. 55:25

     Well, I mean, people used to do, like, [REDACTED:generic_id]-hot encoding where your embedding layer would be the same size as, like, your vocabulary and I, I think that, like, a denser representation does work better.

491. 55:34

     So I think there's, like, a sweet spot in, somewhere in the middle. I, I don't think as you get larger it's always better.

492. 55:39

     I agree on that. I think someone needs to do a research paper. Yes. That should be a new research top- ... I, I've never seen this paper before, so, like, that would be very interesting.

493. 55:48

     So yes, question?

494. 55:50

     I had a question for the tokenizing, uh, questions as well. Is there any reason, or, sorry, is there any reason why you pad unused tokens like Gemma did, or Gemma models specifically with the tokens?

495. 55:58

     There's a bunch of-

496. 55:59

     Yes

497. 55:59

     ... unused tags. What's the reason behind that?

498. 56:01

     The reason for that is it just makes training faster [laughs] sometimes. So depending on ... So there was ... I think Andre was the [REDACTED:generic_id] who tweeted about depending ...

499. 56:10

     If you pad the token, if you pad the vocabulary to a specific number, you can actually make training faster because in the NVIDIA GPUs, when you use tensor cores, if you pad it correctly, it can get the data and cache it more appropriately.

500. 56:23

     So like, for example, the cache size is, like, 64. I think it's 64. Okay, maybe I'm making stuff up. But, like, essentially you have to pad it to a multiple of 64, something like that, and so sometimes that happens.

501. 56:32

     Another [REDACTED:generic_id] is, like, some people want to add ... When you want to do more fine-tuning, when you want to, like, train the model for more, you want to use [REDACTED:generic_id] of those unused tokens for your own purpose, and so they left some holes in there.

502. 56:43

     Um, yeah. Does that kind of answer your question?

503. 56:46

     I was wondering if it maybe caused any weird issues with the model if you hit those on other occasions or ... But-

504. 56:51

     So when you do tokenization, assuming you don't encounter these tokens, you won't have any problems. But if you do,

505. 57:00

     then there are problems, yes. Um, so for example, if you do Llama 3 fine-tuning, if you use the base model for Llama 3 and you accidentally use [REDACTED:generic_id] of those tokens, you will get NaNs for, um, for fine-tuning, right?

506. 57:13

     So you have to be very, very careful. And so, like, I think what we did is for Unsloth, we actually, we actually find these untrained tokens fast, set them to the mean of all the embeddings, and you won't have these issues.

507. 57:24

     Um, so I think that's actually a model creator's problem, is, like, they probably should have not set it to [REDACTED:generic_id]. Um, I don't know why they did that, but anyways, um, yeah.

508. 57:32

     They should have set it the, like a, you know, normal distribution or, like, some, you know, just random initialization. Um, yes. Does that kind of ... Yeah, okay. Any other questions?

509. 57:40

     Okay. Oh, yes.

510. 57:42

     Yeah, just a quick question. Uh, why was there an UOS and not a VOS at the beginning? Like, what if I want to ... The AI would ... Uh, you know, you want to learn that hello is, like, most sentences that they start with hello, so why don't you have a beginning of sentence token?

511. 57:56

     Oh, yeah, yeah. You can put a beginning of sentence token. I just didn't do that. Um, you should put a beginning of ... Most language models would put a beginning senten- ...

512. 58:03

     Like, what is ... You know. I put the end of sentence. You should probably put a beginning of sentence as well. That's actually very important as well. Um, most models do that now.

513. 58:12

     Um, they find that to be very helpful. To be honest, I don't think so it's actually that effective. I think the beginning of sen- the BOS token came from the old style, um, um, the old style, the CLS token for, um, I think it was the first token, for, like, BERT style.

514. 58:27

     Um, so, like, they had the classifier token at the very start. I think it was at the very start. I'm not 100% sure. But I think that's where it came from, the big ...

515. 58:33

     I, I don't think so the beginning of sentence token actually makes that much difference, um, but you should put it. You should put it. You know, giving the model more freedom to move is always better.

516. 58:43

     Um, so yes, I probably should have put a beginning of sentence, but, you know, I, yeah. For demonstration I did not do that. Um, yes. Okay, we did that.

517. 58:50

     Right. So, like, the green [REDACTED:generic_id], um, right, so, like, the, the attention block is kind of encoding the stuff that we described, right? Predicting the next word based on the previous words, right?

518. 59:03

     And so, like, the attention block is that first part. The MLP block is just the mixing pump component. Um, and this is kind of the transformer architecture kind of, like, in visualized, and you just repeat this L times.

519. 59:14

     Um, and that is a transformer. Um, now the, another question I always have is why is training language models not O of N cubed? Because, like, aren't you, like ...

520. 59:28

     Given the word hello, you're predict- predicting my, right? And now we have hello my, you're predicting name. And then you have hello my name, and you're predicting is, and so on, right?

521. 59:37

     Shouldn't this be the training data? Why is the training data just hello my, my name, name is, is Daniel, Daniel Yang, right? This is the training data that you actually see.

522. 59:48

     Why is it not this? Can ... Does anyone know why?

523. 59:53

     Complexity?

524. 59:55

     Sorry?

525. 59:56

     The complexity.

526. 59:57

     The complexity?

527. 59:57

     Yeah, so it will be, uh-

528. 59:59

     Yes, the complex- yeah, very bad complexity, right? So, like, if, if the sentence is, like, [REDACTED:generic_id] ... Okay, how ... If the sentence is, like, 100 words,

529. 1:00:07

     what do you think? How many ... Quite bad, yes. Basic [REDACTED:generic_id]. Yeah. So, like, a [REDACTED:generic_id] plus [REDACTED:generic_id] plus three plus four plus five all the way to plus 100, right?

530. 1:00:16

     So, like, N divided by [REDACTED:generic_id], [REDACTED:generic_id] plus 100, I think. I can't remember my maths, but yeah, something like that. So it's N ... Yeah, it's very bad. Um, and that's if you have [REDACTED:generic_id] sentence.

531. 1:00:25

     If you ... What happens if you have, like, 10 sentences? Oh, my. Yeah. But, like, do, does anyone know why language models don't need to do this? Like, we don't actually need to do this, right?

532. 1:00:34

     So, like, we can skip ... Essentially, instead of, instead of having this as the training data, your training data is simply my name is Daniel, and, and shift it by [REDACTED:generic_id] up, and that's your training data.

533. 1:00:44

     Why is it not this?

534. 1:00:45

     That changes the show, the encoding of-

535. 1:00:48

     Oh, yes, we haven't talked about position encodings yet. Yeah. Okay, but you actually don't need position encodings.

536. 1:00:54

     Attention.

537. 1:00:54

     Oh, okay, yeah, attention. What ... Oh, yeah, the attention mechanism.

538. 1:00:57

     Yeah, because of attention.

539. 1:00:58

     Oh, correct. That's the answer. Yes. It's because of attention. Well, actually specifically, um, masked attention. Right, so that's, that's the trick. Okay, we'll be talking about that. Um, we've been talking about it a few times.

540. 1:01:11

     Um, and- I'll give you the code again. Well, actually, the maths formulas for transformer architecture, right? So like attention block. Um,

541. 1:01:20

     we will be now talk about the attention block, right? So like the Z is equal to the softmax of QK transpose over r- um, root H plus MV. Um, and as, as you mentioned, it is the attention mechanism which allows us to skip the O of N cubed complexity and make it O of N squared.

542. 1:01:39

     Why? Because remember, we want to mask out future tokens because we don't want to predict on future data, right? So like by using this mask, weirdly, this mask allows you to train more efficiently.

543. 1:01:51

     Um, you know, it's funny because like attention is O, O of N squared, so the longer your sequence is, the worse the complexity. But actually, there is a special trick which you use a mask, and this actually makes attention not that bad.

544. 1:02:05

     Um, so instead of doing hello to predict my and so on, so on, so on, the attention mask acts as this methodology, right? So the attention mask itself acts as, um, you don't need to do like all the complicated, you know, all of the words, predict the next word.

545. 1:02:23

     Um, okay, this is... Okay, probably should have... Um, so we, we'll be now talking just about the attention itself, right? So like softmax QK transpose over root DV. Um, just a reminder that whenever you see QK transpose, the query and keys, um, I do not like to, like, there's like these explanations like what is a query, what

546. 1:02:39

     is a key. I do not like that actually approach. I would like this to be a maths approach. Um, so my view is given the matrix X, which is your embeddings, right?

547. 1:02:48

     So remember, hello is a vector of numbers, right? You multiply this by some weights, WQ, WK, and WV, and you get back QKV.

548. 1:03:00

     Q is query, okay, keys and values, but, uh, um, that's a very vague interpretation. I don't really believe... Like, I don't really trust those interpretations. It's not that clear.

549. 1:03:08

     Um, just assume it's just maths, okay? Just like get your X matrix and multiply by weights, and you get some extra weights. That's my view. Um, and so that is kind of...

550. 1:03:18

     So, like, if you see why I stacked it like this, does anyone know why I stacked it like this? Like, why did I present it like this specifically? Why is it, like, the presentation like this?

551. 1:03:28

     Any h- any points? You want to do the composition. Sorry, what? You want to do the composition. Composition? Decomposition. Decomposition. Yeah. Interesting. Okay, that's a very interesting point. Um, but no.

552. 1:03:40

     You're just lining up the dimensions of the multiplication. Correct. I just... Yes, that's correct. So I just lined it up such that it's easier to see, and if you take the matrix X and you multiply by WQ- WQ, you'll get Q, right?

553. 1:03:52

     And this is actually the correct maths, um, dimensions and stuff like that. Um, and so like I li- I like to normally tell people to, like, visualize transformers as maths.

554. 1:04:02

     It's actually, in my view, it's easier. Okay, but I'm not sure for other people, but my view, it's easier. I do not like it when they say, "Oh, queries, and, like, you're trying to do keys and values."

555. 1:04:11

     I don't even know what that even means. Anyways, um, and the yellow components are the ones you want to train. X is what you want to train. WQ is what you want to train.

556. 1:04:19

     WK and WV. And QKV are just the components afterwards.

557. 1:04:27

     When you have the... So remember, you have the Q, you have the K. All you need to do is when you do K transpose, you transpose the matrix, and you do Q times K transpose, and you get this big square matrix called QK transpose, right?

558. 1:04:41

     Hello, my name is Daniel, and so on, right? So like that, that's kind of what I want to visualize is, like, you know, it's actually a... When you do QK times K transpose, you get a square matrix.

559. 1:04:51

     Um, and all you need to do now is do the softmax divided by root D, right? So softmax, essentially each row you normalize to [REDACTED:generic_id], right? The sum of the exponentials must be, right-- You need to, like, normalize them.

560. 1:05:04

     Uh, do-- Does anyone know why you should do that? And why should you use softmax?

561. 1:05:12

     Any clues? Why do... Yes. To sum to [REDACTED:generic_id]. Yes. Okay. That's the answer, yes. [laughs] But, like, why? Like, why?

562. 1:05:22

     Uh, because when you multiply them, you can get

563. 1:05:25

     nan. Sorry. When you multiply them, you can get nans? No, you can get... out of nan, you get loads of exponentials. Oh, yes. Very good. Oh, that's... Okay. Do you know how to fix that?

564. 1:05:34

     Well, that, that's the point of the softmax, isn't it? Close. Okay, then I'm out. You have to minus the maximum of the row. That's how you fix it. Um, yes.

565. 1:05:42

     Yes. We're going to sample from that distribution. Oh, yes. Very good. Okay. Yes, we want to sample from the dist... Okay, sample from that distribution. But what happens if you don't do the softmax?

566. 1:05:51

     Doesn't this still work or not? Like, what happens if you just do QK transpose over root D? Remove the softmax. Like, why do I have to do softmax? Yes.

567. 1:06:00

     Because then you get exploding gradients. Interesting. Then you can fix that with, like, minus max of the row as well with exploding. Anyone else? Maybe a non-linearity, right? Okay.

568. 1:06:12

     What happens if you don't have a non-linearity then? I'm trying to follow. So does it have to be softmax? Can it be something else? Theoretically, it could be something else.

569. 1:06:20

     Yes. It could be. Yes. That is another er- active area of research which people should focus on, which is, like, why do we need to use softmax? Um, generally speaking, research papers show that is actually the most accurate.

570. 1:06:31

     Um, if you use other activation functions, they might actually not be that accurate, right? So, like, um, but this also is the bottleneck of transformers, is because it's a softmax, it does the row sum of the exponentials.

571. 1:06:44

     Um, this means that you can't actually decompose this, right? You can't actually bring the matrix, matrix multiplications out. Um, and so if someone can find ways to make this faster, you know, you'll get, like, millions of dollars.

572. 1:06:56

     Um, okay, maybe, like, much more than that, but, um, yes.

573. 1:07:00

     And V is just... Remember, the V comes from here, right? So we just take the V, multiply it up again, and we get this matrix at the very end.

574. 1:07:09

     And that is, right-- Oh, yeah, that, that is the final component, right? This, this empty box is what you get out from the attention mechanism.

575. 1:07:18

     For the layer norms, um, t- I don't really wanna explain too much, but the layer norms essentially you take the, um, you take ... You square all the elements per row, you sum them, you divide them by the square root, and you take the mean and then just do [REDACTED:generic_id] divided by it, right?

576. 1:07:32

     All this does is just normalizes the rows to make it easier for the language model to learn, right? So, like, why do people do layer norm? It just makes training easier.

577. 1:07:40

     Um, it's more stable. There's no other, like, there's no other, like, point. There are, like, some theories of, like, you know, batch normalization, like, you know, um, you n- out of distribution you want to make, like, shift towards the distribution of, uh, out of dis- distribution data.

578. 1:07:53

     I just like to think of this as a optimization method. Um, layer norms just make a tr- e- training easier and more stable.

579. 1:08:02

     Um, and layer norm is simply, remember as I said, is you take the X matrix, you do a row sum of all the squares and you take the mean, and then you just divide it, and then you multiply by some weights.

580. 1:08:14

     It's a vector of weights, and that's just layer norm. Um, don't worry too much about, like, what is layer norm or what it does. It just does training better, more stable.

581. 1:08:22

     Um, please add as many layer norms as possible. Um, yes, add everywhere. Layer norms everywhere, um, and you'll make training much better. Um,

582. 1:08:32

     okay. I probably ... Okay, I don't know if you can see this, but in Triton, right, in order to write Triton code for the layer norm, this is the forward kernel.

583. 1:08:40

     Um, we will not be talking about Triton today, but, um, it's actually not that complicated. If you read more intensely, um, ignore all of the, like, components, there is only very few lines for the layer norm.

584. 1:08:53

     Um, it's actually not that complicated. Um, the rest is simply just how to load the data. Um, it's actually not that hard. Um, yeah. The backward kernel is when the problem comes.

585. 1:09:03

     Um, how do we actually do the differentiation of the layer norms? Right, remember, you have to train the W, right, is yellow. Um, you actually have to train that.

586. 1:09:13

     Um, how do we find the derivatives of the W? Um, it is very complicated. Um, and if you want to learn in your own time, you can have fun learning the derivatives.

587. 1:09:25

     Um, it is extremely complicated because there is, like, sums, there is, like, um, you know, row sums. How do we do the derivative of a row sum? Um, it get, can get quite complex.

588. 1:09:36

     I wanted to talk about backpropagation today, but I thought, like, it's probably too heavy maths. Um, so no backpropagation, but we'll be showing ... But I do have, um, tutorials on that, so if you go to the Triton tutorial, um, I followed that.

589. 1:09:49

     That's actually quite helpful. Um, and the backward kernel is just very,

590. 1:09:54

     very problematic. Um, now up to the RoPE embeddings. Why do we do RoPE embeddings? Does anybody know what is a RoPE embedding?

591. 1:10:04

     Yes.

592. 1:10:05

     Uh, it's the ... Well, it's a way to extend, I mean, context, right? When the, the, the

593. 1:10:12

     So you could use the RoPE embeddings to extend context, yes.

594. 1:10:17

     Okay. Uh, that's not what-

595. 1:10:17

     Do you know how? How does it extend context?

596. 1:10:20

     So, well, how does it work-

597. 1:10:22

     Well-

598. 1:10:22

     ... versus YaRN or, like, or-

599. 1:10:24

     How would you use RoPE embeddings to extend context? How, what would you do?

600. 1:10:27

     How would I, how would I do that?

601. 1:10:29

     Mm-hmm.

602. 1:10:29

     I would, uh, create a basically ... What I would do is I would basically create kind of a, um-

603. 1:10:38

     You just multiply the base by [REDACTED:generic_id] and you get [REDACTED:generic_id] times longer context. You multiply the base by 10-

604. 1:10:42

     Yeah, but the problem is if I'm looking for, like, 1 million context, right?

605. 1:10:46

     Mm-hmm.

606. 1:10:46

     That then the model, well, part of it is trained at, like, 400-

607. 1:10:49

     Correct

608. 1:10:50

     ... or sorry, 4K, right?

609. 1:10:51

     Yes.

610. 1:10:51

     So that's where YaRN might kick in-

611. 1:10:53

     Yes

612. 1:10:54

     ... instead of, like, uh, any-

613. 1:10:55

     So is that the dynamic, dynamic-

614. 1:10:57

     Well, the static or dynamic-

615. 1:10:59

     Yeah

616. 1:10:59

     ... either way, right?

617. 1:11:00

     So how would you solve the problem if it's like you wanna train ... If you wanna have 1 million context length but your data set is only 1,000 words, how would you solve that problem?

618. 1:11:08

     How would you think of solving that problem? Because, like-

619. 1:11:10

     So, um-

620. 1:11:11

     ... some people have said they do 10 million context length. Is there any data sets which is 10 million tokens? Um, how would you-

621. 1:11:17

     There's 15 trillion tokens fine wine, so we have that.

622. 1:11:21

     Oh, no, no, but that, that, that's 15 trillion tokens for, like, the data set. I mean, like, how do we do long context? Remember, when you do long context training, you have to have a document which is at least 10 million words for it to learn how to predict the 10 million plus [REDACTED:generic_id] token.

623. 1:11:35

     So, um, how I would solve the problem would just be to gather better and more diverse data sets, but-

624. 1:11:42

     Yes, that's, that's the-

625. 1:11:43

     I mean, that's the ideal

626. 1:11:44

     ... so what happens if there is a da- there is no data set which is 100 million tokens?

627. 1:11:49

     Then what would you do?

628. 1:11:51

     If there is no data set that's 100 million tokens, I would co- I would, I would synthesize ... I, I, I, I'm-

629. 1:11:59

     How would you synthesize if the model ... It's like a chicken and egg problem.

630. 1:12:03

     Huh?

631. 1:12:03

     How would you do synthesis?

632. 1:12:05

     So, no, no, no. I would, I would basically, um, just create ... I would basically use, like, uh, Claude or, like, any of the state-of-the-art models with, like, a LoRA, and then get ...

633. 1:12:18

     and then basically turbulate the data-

634. 1:12:20

     But are they trained on 10 million tokens?

635. 1:12:21

     Huh?

636. 1:12:22

     If the model itself wasn't trained on 10 million tokens-

637. 1:12:24

     No, so, so-

638. 1:12:24

     ... does it do long context?

639. 1:12:26

     So, so if I was to try to s- to solve this problem for, like, a client-

640. 1:12:30

     Mm-hmm

641. 1:12:30

     ... for example, like, let's say their code base is in 10 million tokens or, or, you know, and they want a 10 million something context or whatever-

642. 1:12:37

     Mm-hmm

643. 1:12:37

     ... then I would, um, basically cr- like, uh, create a, like, a per-

644. 1:12:44

     Syn- synthetic data set?

645. 1:12:45

     S- not synthetic, but a derived data set from what we have.

646. 1:12:49

     Okay. Interesting.

647. 1:12:50

     So assuming we do not have ... But I can't assume that we have no data, right?

648. 1:12:55

     Hmm.

649. 1:12:55

     So, like-

650. 1:12:56

     Good point. Okay. I don't know. I, I think it remains to be seen. Like, many claims by companies, 10 million context, 100 million context. I question, question, question, question, question.

651. 1:13:05

     Um-

652. 1:13:05

     Well, I, I've only seen 1 million actually work, so I mean, but-

653. 1:13:08

     Yeah

654. 1:13:09

     ... and that's ring attention in theory, right?

655. 1:13:12

     Okay. Okay, now we're going into ... Okay, yes. Okay.

656. 1:13:15

     Sorry. Apologies again.

657. 1:13:15

     Yeah. Okay. No, no, no, that's fine. I was asking the questions, but ... Okay, wait. The question was, like, what is a RoPE embedding? Um, someone did mention, like, positions.

658. 1:13:22

     What does that actually entail? What do you think is the point of a RoPE embedding?

659. 1:13:28

     All it does is you want to tell the model to learn what is the position of the words, right? So like, "Hello, my name is Daniel," it actually has a meaning, like hello is like the first token, right?

660. 1:13:41

     But then if you put hello as the third token, what's the difference? There is a difference, right? So like depending on where the word is in the sentence, it matters.

661. 1:13:50

     So the whole point of embeddings, RoPE embeddings, is it lear- it tells your model to learn where is the position of this component. Um, and old style, they use, um, absolute, like relative, like, you know, absolute positions.

662. 1:14:06

     Um, RoPE embeddings does, like some special tricks like, you know, times a cosine plus a si- um, times a sine and does some sort of like special rotation and stuff like that.

663. 1:14:14

     Um, the paper found that if you do RoPE embeddings, it actually has high accuracy. Um, and, you know, everyone does RoPE embeddings now. Um ...

664. 1:14:23

     So-

665. 1:14:24

     Yeah.

666. 1:14:24

     Why do we need to, uh, copy all cases then if we already have the positional information in the embeddings?

667. 1:14:32

     You mean lowerca- sorry, the posit-

668. 1:14:34

     At the very beginning you mentioned that there is a lowercase version of BERT-

669. 1:14:38

     Yes, there is

670. 1:14:39

     ... or

671. 1:14:39

     I think BERT did not ... I don't know, did, did BERT use RoPE? I don't think so. BERT used absolute-

672. 1:14:44

     Um.

673. 1:14:44

     Yes, that's the problem. I think BERT used absolute, I think. Um, I don't remember anymore, but ... Oh.

674. 1:14:50

     RoPE didn't exist at that time.

675. 1:14:51

     Oh, yes, yeah, exactly. So RoPE did not exist, yeah. And so like this paper, the Roformer paper shows, so previously people used absolute position encodings, which simply just adds a position.

676. 1:15:00

     Like you can literally just add, like if the position is [REDACTED:generic_id]- a [REDACTED:generic_id], just add [REDACTED:generic_id]. If the position is [REDACTED:generic_id], add [REDACTED:generic_id]. If the position is [REDACTED:generic_id], just add [REDACTED:generic_id].

677. 1:15:08

     That's, that's literally what they do. Well, actually, well, not, not exactly, but like, you know what I mean, right? You have to divide it by some sort of normalizing factor, right?

678. 1:15:15

     If the position's [REDACTED:password],000, don't add [REDACTED:password],000, right? You would like destroy training, but that's kind of what they do. Um, and what they show is if you do RoPE, you can essentially increase accuracy somehow.

679. 1:15:27

     Um, and we just use this as gospel. We just treat this as true and everyone uses RoPE now. Um, yeah. Yeah.

680. 1:15:37

     In that case, do you have an opinion on YaRN versus RoPE versus ... and YaRN dynamic-

681. 1:15:42

     So YaRN is kind of RoPE. So YaRN just does ... I, I'm assuming YaRN is r- is RoPE, but it does, it does like ... Actually, d- I, I don't think I should comment on this because I'm not an expert on that.

682. 1:15:52

     Okay.

683. 1:15:52

     Doesn't it does like, it does ... Yeah.

684. 1:15:55

     I mean, not as ... So I'm no expert on, obviously, I-

685. 1:15:59

     I'm not expert on long context

686. 1:16:00

     ... on RoPE versus YaRN, but since BLM only supports static YaRN, unless you're planning on trying to go to 1 million context, it's actually kind of terrible. So-

687. 1:16:13

     I ... Is YaRN the [REDACTED:generic_id] which it does like the base, like randomly changes? Like if you have position, if you have like up to 1 million context and you do 1 million and [REDACTED:generic_id] context, the base changes with that, like the factor changes.

688. 1:16:27

     Is that YaRN?

689. 1:16:28

     That, that's-

690. 1:16:28

     Dynamic changing. Yeah.

691. 1:16:29

     So, and that's the issue is that when short, in like let's say five shot, right? That's not even ... Or a short five shot, right? That's not anywhere close to 1 million context in, in theory or maybe like let's say a three shot or [REDACTED:generic_id] shot.

692. 1:16:43

     Um, like so-

693. 1:16:45

     Ta-da.

694. 1:16:45

     And, but obviously dynamic YaRN in theory could fix this RoPE issue-

695. 1:16:51

     Uh-huh

696. 1:16:51

     ... where like we just take this RoPE as gospel where-

697. 1:16:57

     Okay.

698. 1:16:57

     Um, y- but are you following kind of my question here?

699. 1:17:00

     Yes. I, no, I think dynamic YaRN is just RoPE though. Like, okay, that's weird. Okay, maybe I'll unplug and re-plug. So like the screen kind of went away. Let me just read this, do this again.

700. 1:17:10

     Is this like ... Is it, is it, is it not, not working or is it like screen or

701. 1:17:19

     no screen? That's weird.

702. 1:17:23

     Can you a- answer questions while we're at it?

703. 1:17:25

     Oh, yes. Okay. Yes, I'll answer some questions. Um-

704. 1:17:29

     Great.

705. 1:17:29

     Yeah. Anyone have, anyone else have questions? Wait, okay. Wait. I need to refresh the ... Oh, if anyone has like take a break, you can take a break now if you want.

706. 1:17:38

     Um, and if you have like other questions ... Yes. Okay, question.

707. 1:17:42

     Um, so I don't know Ya- what YaRN is, what seems from the discussion is like, uh, practice in the context window of like

708. 1:17:50

     RoPE and like YaRN.

709. 1:17:51

     Sor- sorry. YaRN is-

710. 1:17:52

     So, yeah, so I, I don't know what YaRN is, but I was hearing some stuff about like extending the context window and I'm just wondering that like, uh, what is the issue with like RoPE when it comes to like the, the context window?

711. 1:18:06

     Why isn't it like can't just have like a infinitely, like a, a, very big context window? Is there a problem in like doing that?

712. 1:18:13

     So, so the question is for context windows-

713. 1:18:16

     Like what, what is, what improvement is like YaRN on like RoPE? Like what is the-

714. 1:18:20

     Oh, okay. What is the improve... Okay. So, so the point of YaRN, like what we were talking about is like how do we make a language model learn ...

715. 1:18:27

     How do we make it do long context without training on long context? Kind of. Yeah. And so like what YaRN does is you can essentially extend the context window automatically by dividing the base of the RoPE embeddings.

716. 1:18:41

     You change the scale factor. Um, I, I don't actually have slides for this, but, um, it's just a methodology which allows you to scale the factor and you essentially magically make the model learn new context, long context.

717. 1:18:54

     Um, that's kind of what YaRN does.

718. 1:18:55

     And you originally wouldn't train on the long context if you were to do this method?

719. 1:19:00

     Yes. It's extremely ... If you do 1 million context length, then your O of N squared is 1 million squared.

720. 1:19:04

     Yeah.

721. 1:19:04

     Which is horrible. Um, there are like some other methodologies like, you know, they want to do like linear transformers and like, you know ... Um, yes, I guess you could try that, but I don't suggest that.

722. 1:19:14

     Um, yeah. So hope ... Yeah. Yeah, sorry. Question.

723. 1:19:17

     I would be very interested, Daniel, on your comment, uh, for the fine-tuning versus in-context learning from low-level perspective.

724. 1:19:26

     What is the difference between in-context learning and fine-tuning? Or what's the benefit?

725. 1:19:32

     That is a good question. So fine-tuning changes the grade, uh, it ... You have gradients for fine-tuning.

726. 1:19:37

     But-

727. 1:19:37

     So up... Yeah.

728. 1:19:39

     But less interested in explaining how it works-

729. 1:19:41

     Yeah

730. 1:19:41

     ... because, uh-

731. 1:19:42

     My f-

732. 1:19:43

     In your opinion, what, what would you do when

733. 1:19:46

     I think it depends. I think I would do in-context learning first, so if you have, like, few short prompts, you shove it in to see if it works. But I would still resort to fine-tuning if you want to be more efficient.

734. 1:19:56

     And if your model doesn't seem to be learning, then you have to go to, back to fine-tuning. There was a paper which was released yesterday, I think. Was it yesterday?

735. 1:20:02

     Um, it showed that in-context learning is very useful, um, and it, like, learns kind of, like, how to be, like, a random forest or, like, a tree. I, I don't f- I think it was yesterday, the paper.

736. 1:20:11

     That was quite useful. Um, very interesting. Um, but I think fine-tuning is still very important, especially if your model is not learning anything and it doesn't seem to be working, then you have to, like, use fine-tuning to change the behavior.

737. 1:20:23

     Um, I don't really have a comment on this. Like, I just feel like you should do everything, um, and try everything. Um, yeah. Any other questions?

738. 1:20:32

     Well, I, I have a couple on the app. I don't know if, uh-

739. 1:20:35

     Oh, did you... Okay. I think my app kind of glitched. [laughs]

740. 1:20:40

     Do you want me to read them off [REDACTED:generic_id] by [REDACTED:generic_id]?

741. 1:20:42

     Oh, uh, maybe I'll t- ... Okay. Wait. Yeah, wait. Okay.

742. 1:20:45

     I'm sorry, I ask a ton of questions.

743. 1:20:46

     Okay. Okay, maybe ... Uh, uh, okay. Let's just ... I'll continue, and then we will do the questions.

744. 1:20:50

     No, no. It, it, it's back.

745. 1:20:51

     Yes.

746. 1:20:51

     So we're good.

747. 1:20:52

     Yeah. Okay, I probably have to... Okay, I didn't ... Okay, I did not see time. I've been more speaking to... Okay, anyways, that is the RoPE kernel. Um, and it might look horrifying.

748. 1:21:01

     It's actually not that bad. Um, it's literally just the formula that did a Q times cosine plus Q times a rotation matrix times sine, and that is just RoPE.

749. 1:21:12

     It's actually not that hard. Um, m- just the code is a bit more annoying, but it's just, like, moving the data. It's just data moving and data, moving data and stuff like that.

750. 1:21:21

     All of the code is just related to data movement, so not that complicated. Um, the most complicated part is the derivatives for RoPE. Um, and you have to use something called rotate half, which essentially rotates half of the...

751. 1:21:36

     So, okay, just read the code. It's, like, minus X2 concatenated with X1. So you're, like, you essentially take the matrix X, you divide it by [REDACTED:generic_id], you take the first half, you put on the second half, and you put...

752. 1:21:46

     You switch the ordering, and you... The first half becomes minus. And the code generally is reasonably well, uh, hopefully, um, for understanding. But the question is, and so you, like, this is Hugging Face code, right?

753. 1:21:57

     So, like, Q times cosine plus rotate half Q times sine. The question is, how do we actually take, find the derivatives of this? Um, this was actually very, a very complicated phenomenon because I, I could see many implementations not doing this correctly.

754. 1:22:10

     Um, and it is very special, the derivative. Um, simply, if you notice, is rotate half the, the function is literally a matrix multiplication, right? It's Q times R, where R is a rotation matrix, and the rotation matrix is minus identity and identity, um, and zeros on the diagonal.

755. 1:22:31

     Um, and if you do this, if... because it's a matrix notation now, simply the derivative is the transpose. And so if you do the transpose, it, the minus sign just flips.

756. 1:22:41

     Um, and if the minus sign flips, it's literally the same as your previous example, just for the minus. Um, okay, I probably, like, w- explained this too quickly. Um, but the point is, if you do matrix mul- multiplication, you can derive derivatives very simply.

757. 1:22:57

     My suggestion is shove the derivatives in Wolfram Alpha or, like, you know, your favorite tool. You can use ChatGPT as well, and you will get the derivatives back. But you must put it in the form that's useful for the computer to see.

758. 1:23:11

     Um, now we'll be talking about the MLP component. Right, so we completed the attention. We completed the RoPE, the layer norms. The MLP is just a mixing component, right?

759. 1:23:22

     So it's a activation function times a, times some weights, you know, multiply some of the o- other weights and stuff like that. Um, all it does is it mixes the signals to make it, like, you know, more fancy.

760. 1:23:32

     Um, you d- in theory, you don't actually need MLP component. Um, like most attention, you don't actually need this part, but you must put it for the model to have more freedom to learn.

761. 1:23:42

     Um, and you know, the famous paper, G- um, GLUE variants improve transformer. Um, you know, very famous author, I'm assuming p- most people know him. Um, but he showed that, um, if you add GLUE, um, SWIG-GLUE, and all these other variants, um, you can actually increase accuracy.

762. 1:24:02

     Um, once again, this is just treated as fact in the machine learning community. Um, we should be doing more experiments, um, than just using this methodology, but, you know, we just treat it as fact, and this is in Transformer Plus Plus, um, the architecture which everyone uses.

763. 1:24:17

     Um, there is a very big difference though. Um, in GPT-2, in GPT-2, they don't use the GLUE variants. Um, they simply just use a normal MLP, right? So X times the weights up, do some sort of activation function, and then you down project it, so all the weights down, right?

764. 1:24:35

     So that's... But then if you do SWIG-GLUE and these new variants, the GLUE variants, you essentially add this component where you do element-wise multiplication, um, and then you do a gate, a down projection.

765. 1:24:46

     So it's actually very similar to the GPT-2 architecture. You just add an extra component. Um, and so I try to, like... There's also, like, a naming change, like up and down, um, up and, gate and up, like, you know, changes and stuff like that.

766. 1:24:58

     But in general, it's, you can see it's very similar, um, just the extra element-wise multiplication component. Um, yeah.

767. 1:25:07

     But there is, like, a new, um, the Neutron paper, for example... Uh, sorry, the Neutron, the new model by NVIDIA, did not use GLUE, and instead they used squared ReLU, right?

768. 1:25:16

     And they showed that you don't actually have to do GLUE anymore. You can just use squared ReLU, and it seems to do okay. Um, although it remains to be seen if it actually is good.

769. 1:25:26

     But yeah. So, like, they showed that if you do squared ReLU, you can remove this. You can essentially go back to the GPT-2 architecture, right? You don't need to use Llama architecture anymore.

770. 1:25:35

     Right, so Llama and Mistral, Gemma all used the second equation. GPT-2 used the first equation. You can go back to the first equation, but the trick is the F must be very special, and that's called squared regul- ReLU.

771. 1:25:49

     And they showed that if you do this, your accuracy does not degrade that much. Um, and so yes, the paper. And interestingly, if you see [REDACTED:generic_id] of the authors, it's the same author.

772. 1:25:58

     [REDACTED:generic_id] of the author's names is similar. Um, you know, the name is the same [laughs]. So, you know, they were also the ones to showcase the squared ReLU. You don't need to do GLUE anymore, um, and you can simply just use squared ReLU as well.

773. 1:26:14

     There is a research paper which I highly suggest people to read, and it's called The Physics of Language Models. Um, it is extremely long though, um, but it has many nuggets inside, and I highly suggest people to read this.

774. 1:26:27

     Um, they show example... They actually did so many testing, tests and experiments, and they showed that if you used GLUE variants or gated MLP, it actually reduces the model's capacity to learn on small models.

775. 1:26:40

     Okay, so that's the, that's the point. It's on small models. On small models, on small models, if you do GPT-2, the first formula, it does better than if you do Llama, Mistral, Gemma, the second formula.

776. 1:26:51

     Only on small models, right? That's the point. Only on small models. Um, and there are other special things inside the paper which I highly suggest. Um, it's extremely useful.

777. 1:27:01

     For example, they say that if you do mo- if you change, if you do, if you... Like, for example, the activation function, which activation function did you use? Um, should you use the SiLU or GeLU or ReLU or whatever?

778. 1:27:11

     Uh, that's not that important, right? So, like, it's not, not that important. If you use, like, biases, oh, it's not that important. Um, so, like, there's so many different things that you don't need to do, um, the paper shows, but, you know, people just treat it as gospel, "Oh, we have to use this specific component."

779. 1:27:26

     Um, my suggestion is, you know, we should do more testing in the AI space, like, you know, which variants are the most important, but I think this paper's pretty useful.

780. 1:27:35

     Um, the code for, like, the, you know, the SwiGLU kernel is, that's the forward kernels. Again, it's not that complicated to do it. Like, the second part is actually useless.

781. 1:27:47

     It's, the SwiGLU kernel is literally three lines. It's just, it's just the three lines which are commented. The rest is just data loading, how to actually load the data into the GPU.

782. 1:27:56

     The... Or it's not that important. Um, and, you know, if you use Torch.compile now, you can simply generate these kernels automatically, um, and this makes your training much faster.

783. 1:28:04

     So that's what I suggest people to do, um, just use Torch.compile. You don't have to re- rewrite Triton kernels. Yes.

784. 1:28:10

     Um, can you go a few slides back to that, those, uh, the, the equations there and the, these slides.

785. 1:28:18

     Uh, [REDACTED:generic_id] more. Yeah, this [REDACTED:generic_id]. So what's, like, W gate in here?

786. 1:28:22

     Sorry, W...

787. 1:28:24

     W gate.

788. 1:28:25

     W K?

789. 1:28:26

     W gate.

790. 1:28:26

     Oh, gate. Oh, it's just another matrix. So W gate, W up, you train this, um, and W down. These are all, you train them. W down, W up, these are all you train.

791. 1:28:36

     They're just numbers. Um, so, like, matrices, X times W gate is, like, it's... So remember the, um-

792. 1:28:43

     You said the gate operator

793. 1:28:44

     ... uh, this thing.

794. 1:28:47

     Yeah.

795. 1:28:47

     Right. W, this is for attention. W, uh, X times WQ, WK, WV. Assume it's just W gate, W up and W down, um, and it's the same thing. So you just train these.

796. 1:29:00

     Um, does that kind of answer your question or...?

797. 1:29:03

     Um, so is W up, like, a basically, like, just upwards-

798. 1:29:09

     Oh, no, no, no. It's just a naming convention.

799. 1:29:10

     Okay.

800. 1:29:11

     W up is just... It's just a fr- it's just a, it's actually called up projection. The naming convention is, is like WB, WA, WC. It's just a naming convention that people like to use, W up, W gate and W down.

801. 1:29:23

     Um, they do have meanings. So W down is a down projection. Up means up projection. Um, so essentially you take the matrix and you, like, make it larger and then you project back to a smaller version.

802. 1:29:33

     It just makes the model better to, like, you know, make the model has, ha- have capacity to learn. Um, so it's just a naming thing. Um, any other questions?

803. 1:29:43

     Okay. Um, the, as I said, like, before, the derivatives are always a pain. If you do the derivatives of SwiGLU, it is a nightmare to do, and I do not suggest you to do this.

804. 1:29:56

     Um, but I had to manually do all the... You can see all my comments, so, like, the comments are actually there. If you do see more carefully, I wrote it in math formulas of how to actually take the derivatives, and it's extremely painful.

805. 1:30:07

     Um, I highly suggest you not to inflict pain on yourself by doing this. Um, it took me many days to do, so do not. I don't suggest this. Um, yes, question.

806. 1:30:17

     How do you verify what, you know, like these things then? What, what do you-

807. 1:30:21

     That is a very good question. Uh, so I use Desmos. Um, so Desmos is a graphing, graphing, online graphing calculator. You type all these equations in, and then you can see, does the graph align?

808. 1:30:32

     Um-

809. 1:30:33

     No, but along with all the data loading, what if you have made a mistake in the data loading?

810. 1:30:36

     Oh, every single component you have to be careful. So every single component has to, you have to check. Um, oh, is this component u- uh, correct? Is this component correct?

811. 1:30:44

     Check all of them. Um, and so, like, I normally... So, like, you isolate each component separately and then test it. I will, I'll talk about that, actually. Yeah. Um, any other questions, no?

812. 1:30:55

     Yes. Okay. Um, and this is the cross-entropy loss kernel. I'll probably just skip this. Don't have enough time [laughs]. So, um, I wrote this as if you want to inspect the formulas and stuff like that, how do we do the derivatives for this?

813. 1:31:08

     Um, you can do this. It's not that complicated. Um, actually it is very complicated. Um, I did spend a lot of time trying to, like, work out the derivatives [laughs].

814. 1:31:18

     Um, uh, it, it might be a bit foreign for some people for the derivatives. The main reason why it gets complicated is when there's sums, right? For, like, whenever there's sums, I dislike, you know, doing derivatives when there's sums is always painful.

815. 1:31:32

     Um, matrix differential is actually very easy. If you do X times W, the derivative of W is just X transpose. It's very simple. But if you do Derivatives when the sums, ugh, it's horrible.

816. 1:31:44

     Um, yeah, it's quite horrible. What you can do for the sum when you do derivatives, if you transform the sum into a matrix multipl- matrix multiplication. Right, so a sum is just X times a nu- like, a vector of all ones.

817. 1:31:57

     Right, so that's called the row sum, and essentially if you do this, you can actually make differentiation much easier. Um, but I won't be talking about that, um, that's for another topic.

818. 1:32:04

     Um, and someone was talking about stability for softmax. If you minus the maximum of the row, you can make softmax much more stable. Um, and this is to, like, reduce exponentials of large numbers and the, like, essentially it takes over the entire exponential, right?

819. 1:32:21

     And so, like, if you do this trick, when you minus the maximum of the row, this makes training much more stable. Um, always do this. Um, yeah, always do this.

820. 1:32:31

     Um, yeah. And that's the code for the forward, um, not that important. Oh, yes, I wrote the code for the backward as well. Um, oh, I said this is the ...

821. 1:32:40

     Sorry, this is the forward, um, the forward. Um, and it is quite long, but I wrote all of this down for your own leisure if you want to read and implement this, have fun.

822. 1:32:50

     Um, but I wrote this step by step, right? So, like, take Y is equal to log sum of X, you know, then I simplified it out. Like, you know, if you, if you exponentiate both sides, right, you can do exponential of the Y is equal to the sum of the, sum of exponentials and so on.

823. 1:33:04

     All right. And so, like, I wrote this all down. Um,

824. 1:33:08

     for the ... But there is a methodology which we showed in Unsloth, is you can use chunked cross entropy, and this is actually very helpful for large models. Um, your logits are very large, so if you chunk them, you can act- you can make multithreading much better for the GPU.

825. 1:33:24

     And so, like, the problem though is, like, the derivatives, the forward propagation, you have to be careful now when you do chunking. Um, so, like, essentially you divide these into slivers and paralyze each component.

826. 1:33:35

     Um, I also wrote some, you know, maths and stuff like that for you to review. Um, and how you actually do the chunk sum is very interesting. Um, the chunks, the log chunk sum or the log sum exponentials is just a log sum exponentials.

827. 1:33:49

     Um, there's, like, you have to do some manipulation, and it's actually very interesting. Um, you don't actually need to change that much code to make it work. Um,

828. 1:33:57

     okay, now we will be going to the next component, which is to investigate the Llama architecture. Um,

829. 1:34:04

     hopefully this works.

830. 1:34:05

     Just a quick question.

831. 1:34:06

     Yes, question.

832. 1:34:06

     The chunking, is that something that you guys wrote as a part of Unsloth, or is it right now a part of, I don't know, PyTorch or something?

833. 1:34:14

     It is part of Unsloth currently. Um, I think I heard that the PyTorch team will be including this in Torch.compile, although I'm not 100% sure. Um, this reduces memory.

834. 1:34:24

     This just makes, this makes long, large context ... Oh, sorry, large vocabulary sizes work. So the biggest issue why you have to do chunking is CUDA, um, NVIDIA GPUs has a limit, 65536.

835. 1:34:36

     I think that's [REDACTED:generic_id] to the power of 16, or that amount. I think so. Yeah, yeah. So there is a limit, and so if you go, if your vocabulary size is larger than 65536, you must do chunking.

836. 1:34:46

     Um, yeah, you have to do chunking. So, like, if you have a ... Like, I, I think, was it Gemma? 128,000, or the ... I'm getting confused. I think it's 128,000.

837. 1:34:54

     You have to divide it into [REDACTED:generic_id], um, and so, like, your chunks would be [REDACTED:generic_id]. Um, so it should be in PyTorch in their future release maybe. Um, yeah.

838. 1:35:04

     So now we learnt all of this. Now you can read the code for Llama. So if you go to, um, if you go to the code, like modeling Llama.py,

839. 1:35:15

     um, it should be in the slides, but you can also type in Google, like modeling P- uh, you know, modeling py. Um, this,

840. 1:35:24

     right, there is your ... If you go to line 94, you have your RoPE embedding, which we talked about, right? Just assume, don't worry ... Uh, oh, okay, I need to ...

841. 1:35:34

     Okay, I'll probably use my mouse. Yeah. Okay, I don't really like how GitHub does the ... It's kind of annoying sometimes, but I think I can disable it if I log in.

842. 1:35:41

     Anyways, the symbols. Um, so if you go to this,

843. 1:35:47

     right, this is not important. If you go to the first [REDACTED:generic_id], it's line 74, the Llama RMS norm, right? This is the layer norm kernel, right? This is the code for the layer norm kernel, just this much.

844. 1:35:57

     Um, right, this is the ... You take the row, you take the sum of the r- you take the squares of the r- like, each row, you sum them, you divide it by the mean.

845. 1:36:06

     Remember, this is the only reason why you do layer norm, is to make training more stable. Um, and it's actually not that complicated, it's just these few lines. Um, the rest as a ...

846. 1:36:15

     The rest is just bloat code for, like, you know, you have to set stuff up, you have to do random initialization, blah, blah, blah, comments and stuff like that.

847. 1:36:22

     Um, but that's the layer norm kernel. The RoPE kernel, the rotary embedding is a RoPE kernel. This is just setting stuff up. This, right, this is setting stuff up.

848. 1:36:33

     Um, forward, right? This is the most important component, which is the forward component. Now, don't get scared by this. That's because we fixed a bug. So this is actually our bug fix [laughs] that we did.

849. 1:36:44

     So this is all in transformer architectures, is, like, you have to be very careful when you down cast to float16 if you use bfloat16, right? This is actually very important.

850. 1:36:52

     Pe- before, you have to be very, very careful, um, when you use float16 and bfloat16 training, mixed-precision training, because it would down cast incorrectly and your RoPE embeddings will be wrong.

851. 1:37:03

     Um, so it's actually, it's not supposed to look as ugly as this, but unfortunately it looks ugly now. Um, before it was just this, right? But this is just setting up code, um, that, you know, we have to, like, fix the bug and stuff like that.

852. 1:37:16

     Um, yeah. So it's actually not that long, right? So, like, it's act- it's just this, um, yeah, up to here. It's the ... So whenever you see these architectures, it just looks complicated, but it's ...

853. 1:37:28

     Yeah.

854. 1:37:29

     If you do torch.compile on this, uh, YaRN, you'll get similar performance at the Triton kernel that you're seeing, right?

855. 1:37:35

     No, no, no. So there are, like, some ... If you do Torch.compile, we're still, like, I think [REDACTED:password], [REDACTED:password]% faster. So no matter if-

856. 1:37:42

     Faster than your kernel that you wrote?

857. 1:37:43

     S- sorry, if you use Torch.compile plus Unsloth

858. 1:37:50

     Oh, it's, we're still [REDACTED:password] times faster. Also, not [REDACTED:password], [REDACTED:password]%.

859. 1:37:54

     Oh, nice.

860. 1:37:54

     So we're [REDACTED:generic_id] times faster than Hugging Face plus Flash Attention [REDACTED:generic_id], but Torch.Compile, you know, they're adding kernels from multiple packages in. So yes, they could learn from Unsloth and put it in.

861. 1:38:04

     I'm assuming they're doing that. I did have a talk with them, so yes, they're probably doing that already. But yes, we're still [REDACTED:password]% faster. Um, there was like some...

862. 1:38:11

     I think someone tested this last week. Um, yeah. Is that... Any other questions?

863. 1:38:18

     Okay. This is not important. This is for, this is the [REDACTED:generic_id] we talk about for linear scaling, for RoPE embeddings. If you wanna extend the context length, that's kind of what they do, right?

864. 1:38:28

     This is, this is for scaling, not, not that. There is a rotate half, rotate half part which I was talking about, right? How do we actually derive the formulas for the gradient?

865. 1:38:38

     Uh, how do we actually do the RoPE embeddings is just this. Um, the MLP, which we talked about again, remember the MLP layer? Right, so there's a gate projection, the up projection, the down projection.

866. 1:38:47

     Um, this code is bloat again. Ignore this. It's just this. It's just [REDACTED:generic_id] line. That's, the rest is bloat. Um, ignore that, right? So like, the down projection is just the down projection, right?

867. 1:39:01

     The activation function, the gate projection times the up projection, right? So that's, that is the MLP. That is the, you know, squiggly. [REDACTED:generic_id] line, not all of this, and that's just for training purposes.

868. 1:39:12

     Um, repeat KV is just an... That's for like, you know, um, okay, I won't explain this too much, but that's for the attention part. Um, this is just to make inference faster.

869. 1:39:23

     When you do QK and when you go back to slides, um, if you go to, um... Where is it?

870. 1:39:34

     Right. WQ, WK, and WV. Instead of training WK and WV, you train a small sliver and you repeat this, and this can make inference faster. Um, and so like you don't a- we don't actually train the full ma- the matrix size WK anymore.

871. 1:39:49

     We train a small sliver, um, and we just repeat this.

872. 1:39:54

     The attention again, all of this is just preparing, right? This is just preparing, right? And then the... Okay, ignore, ignore, ignore. Blah, blah, blah. Bloat. Get rid of this.

873. 1:40:04

     Um, don't look at that. Um, there is Q, K, and V. That's the matrix part. Um, there is some... The only problem I find that people struggle with is there is like dimension manipulation.

874. 1:40:16

     You have to like manipulate the dimensions of the output. That is actually kind of annoying. I do agree, this is actually very annoying, but just assume it is the this, right?

875. 1:40:25

     That's all we're trying to do is this. Um, and that is just these, these three lines do this, uh, do this. Um, and then we want to do QK transpose, right?

876. 1:40:38

     Or you have to... Sorry, you have to apply the RoPE embedding. Don't forget to ro- apply the RoPE embedding. Um, and then QK tr- that's the repeat KV. Repeat KV is the [REDACTED:generic_id] that, uh, the trick that I said to, uh, make inference faster, um, by repeating the...

877. 1:40:52

     So if you go back here, the K and the V, the K and the V, you take only small slivers. You only train small slivers, and you repeat them four times.

878. 1:40:59

     Um, and it does not reduce accuracy that bad. Um, and that's the repeat KV.

879. 1:41:05

     This is the QK transpose, right? Torch.MatMul QK transpose. This is the attention part.

880. 1:41:14

     Softmax. Dropout. Okay, no [REDACTED:generic_id] uses dropout anymore. Get rid of that line. Um, matrix multiplication, right? So this is... So up to here, up to here is

881. 1:41:26

     QK transpose over root D softmax times V, right? So up to here. And then we do some sort of... And then we have to do some, um, output projection as well.

882. 1:41:36

     Um, the rest... Okay, whenever you see this part, just ignore that, right? Whenever you see like if self.config.pretraining_tp is more than [REDACTED:generic_id], this is just for, um, this is just for faster training, so you can get rid of that.

883. 1:41:47

     Well, not for faster training, for like, you know, training across multiple GPUs. You don't... You can ignore that and just assume it's just that [REDACTED:generic_id] line. Um,

884. 1:41:57

     now there is more code like Flash Attention [REDACTED:generic_id]. Ignore. That's just for faster Flash Attention. Ignore, ignore, ignore. Right. No, you don't need to see this. S- um, scale dot product attention is a faster version of attention that is native to PyTorch.

885. 1:42:10

     Also ignore. You do not need to see that as well. Pretend you didn't see that. Um, and then finally we get to the decoder layer, right? Remember each we show, uh...

886. 1:42:19

     Okay, wait, maybe I should exit the slides. Um,

887. 1:42:23

     where is it? No. So remember the decoder layer. Um, so this, right, remember we said we repeat L times, right? This is repeat L times. This is just shoved in, in the decoder layer, right?

888. 1:42:41

     We call this [REDACTED:generic_id] decoder layer. And again, we do the LayerNorm. Remember, put LayerNorms everywhere. Do LayerNorm, do attention, add some residual. Um, this also makes training more stable, right?

889. 1:42:54

     When you add residual, it makes training more stable. Do more LayerNorm, do more L- um, do an MLP, add more residual, and then we completed, right? That's just [REDACTED:generic_id], that's [REDACTED:generic_id] component of the decoder.

890. 1:43:06

     And remember, we repeat this L times, and the rest of the code is just doing this L times. Now where is the L times? Um, comments, comments, comments. Forward, right?

891. 1:43:16

     You go to forward. Um-

892. 1:43:19

     Sorry.

893. 1:43:19

     Yeah.

894. 1:43:20

     Quick question to the, to the previous [REDACTED:generic_id]. So does it make any difference where you put the layer norm?

895. 1:43:28

     In my opinion, put it everywhere.

896. 1:43:29

     Yeah, but-

897. 1:43:30

     Oh.

898. 1:43:30

     In terms of calculating the residual and then adding it back.

899. 1:43:33

     Oh, that's just by-

900. 1:43:35

     Do you norm after-

901. 1:43:36

     Yes. Why is the ordering? Why is the ordering... Do, I think it's LayerNorm first, then add residual. Is that correct? Or maybe I'm getting confused.

902. 1:43:43

     So it's get residual, then norm, then calculate the attention, uh, unpack the residual, and then there is probably another-

903. 1:43:52

     Yes. The ordering to turn the trick, it doesn't really matter. I think there are some research papers which show if you switch the ordering, it maybe increases, like it decreases accuracy by like 0.01%.

904. 1:44:03

     To be honest, we need to do more testing again. Like, in my view, shove LayerNorms everywhere. Um, this actu- this should make training more stable. Um, but it makes training slower.

905. 1:44:14

     That's the problem. Um, that is the only problem. But I suggest you to sh- put LayerNorms everywhere. Wherever you see what, just shove LayerNorms. Um, yeah, LayerNorms always work.

906. 1:44:26

     Um, okay. And-

907. 1:44:29

     Sorry, just, just [REDACTED:generic_id] more thing then. But then why isn't this an issue that you get the residual before applying the norm and then after, like you calculate attention on that layer after the norm, so you kind of combine the residual before the norm with your calculations after the norm?

908. 1:44:46

     So why do you add the residuals of the previous... So you do, you take the resid- you, you save the state before the LayerNorm-

909. 1:44:53

     Yeah

910. 1:44:53

     ... and then you, and then you do a LayerNorm, and then you add back it in. So why do we have to do that? Is that your question?

911. 1:45:00

     Yeah. Why the, the normalization is between that rather than done before it, like-

912. 1:45:05

     Oh, you could do it before if you want to. You could try it. It depends. I think it's for ac- yeah, it depends. As I said, all of this is like gospel, like, oh, why do we do this?

913. 1:45:15

     It's just people tried it and they said it works. Um-

914. 1:45:18

     I, I, I just meant why this isn't an issue, but I guess that's-

915. 1:45:22

     Oh, what do you mean by not an issue? Like, why does this work or why is this like not a... What, what do you mean by issue?

916. 1:45:28

     Uh, I mean the, the normalization kind of changes the, in some way the representation-

917. 1:45:34

     Yes, it scales it

918. 1:45:35

     ... of the state.

919. 1:45:36

     Yeah.

920. 1:45:36

     Right. So you're combining something from before that change-

921. 1:45:41

     Mm-hmm

922. 1:45:41

     ... with something after that change, and why is it compatible basically? Why it's-

923. 1:45:46

     It's just maths. You just let it, just, it just works.

924. 1:45:50

     Oh, yeah, yeah. I get it.

925. 1:45:51

     Because like, because like if you do that, the autograd engine will still know that you did it, and the derivatives will still be applied correctly. So you just assume that it works.

926. 1:46:00

     Um, to be honest, that is another research topic. You should try that. Um, I'm, I'm being like serious. Like all the... So many research questions that are like open, like why doesn't everyone just put LayerNorms everywhere?

927. 1:46:10

     Like, why don't you put a LayerNorm after the multihead, multi-head attention? Put a LayerNorm after the Squiggler. Put a LayerNorm after the inputs. Put a LayerNorm, you know, everywhere.

928. 1:46:20

     Um, and my, my hypothesis is it makes training more stable, but like makes it slower. Yes.

929. 1:46:25

     In this, in theory, could mechanistic interpretability, um, solve, like, not solve, but explain this?

930. 1:46:34

     What, do you mean like, what do you mean by like explain this?

931. 1:46:37

     Oh, mechanistic interpretability where you take the-

932. 1:46:39

     Do, which method? Do you mean like doing the autoencoder style, like sparse-

933. 1:46:44

     Transformer Lens or like, uh, um, just direct weight, just looking at the weight and what that happened at inference time

934. 1:46:52

     or training time.

935. 1:46:53

     Do you mean like the, uh, what, which method? Like the, um, because I know there's like many me- methods for mechanistic interpretability. Like there's different types of methods.

936. 1:47:00

     Yeah. It's a, it's a-

937. 1:47:00

     Does sparse autoencoder [REDACTED:generic_id] or the...

938. 1:47:03

     Well, Transformer Lens is Neel Nanda's tool.

939. 1:47:05

     Okay. I, I don't, I have not used that, so [laughs] I'm, I'm not that, I'm not expert on that, but-

940. 1:47:09

     So, but you can see the, um, the literal like, um, activations.

941. 1:47:16

     Yes, yes, the a- the activation [REDACTED:generic_id]. Yeah.

942. 1:47:18

     Yeah. So basically what... So I, I mean, I'm gonna not divert from this, but, um, so basically can, do you think we can figure out why this works this way?

943. 1:47:30

     Like, because you said it's a kind of an open question, right? But do you think that using tools like Transformer Lens where we can look at training activate or not activations, but like steps-

944. 1:47:42

     Hmm

945. 1:47:42

     ... where in fact we would have to like, um, I, I'm not sure if I'm explaining itself right correctly, but like do you think mechanistic interpretability is a path to understanding this?

946. 1:47:55

     Good question. Um, could mechanistic interpretability... Okay. Hmm. It depends.

947. 1:48:02

     I think my view is it just LayerNorms,

948. 1:48:05

     if it's specifically on the topic of LayerNorms, it just makes training more stable. I don't think so it has any like meaning, like that's my view.

949. 1:48:13

     Okay. That's fair.

950. 1:48:14

     Like I think like the math equations don't show that it has any meaning. I just find it to be just stabilized training. Um, there was like papers like the, um, uh, what was the [REDACTED:generic_id]?

951. 1:48:21

     Um, batch normalization. I forgot what, what the term was. Um, yeah, like there was like a theory which shows that batch normalization, um, reduces problems of out of distribution data and stuff like that.

952. 1:48:32

     Um, oh, it reduces internal covariate shift or something. Um, that was the phrase which, yeah, um, I don't even know what that even means. But anyways, um, does anyone know what that means?

953. 1:48:44

     There was like a, um, there was like a video for that as well. The, yeah. Does anyone know what that means or... Yes.

954. 1:48:52

     Um, what do you mean by making training more stable?

955. 1:48:56

     So LayerNorms, my view of LayerNorms is when you do-- If you don't do LayerNorms, if you keep... Okay, let's say you take a number [REDACTED:generic_id], you multiply by [REDACTED:generic_id], you get four.

956. 1:49:07

     Remember there's 32 layers, right? If you multiply by [REDACTED:generic_id] continuously, you will get infinities in the end, right? 'Cause you like go out of scope of the float 32.

957. 1:49:15

     So what LayerNorm does is it sh- makes your number go back to a good scale. So if you do [REDACTED:generic_id] times [REDACTED:generic_id] is four, let's divide it by four, go back to [REDACTED:generic_id], right?

958. 1:49:26

     And so now it's [REDACTED:generic_id] again. If you times [REDACTED:generic_id], it's [REDACTED:generic_id] again. Let's divide it by [REDACTED:generic_id] again, go back to [REDACTED:generic_id]. So all LayerNorm does is it makes a scale go back to a good scale.

959. 1:49:34

     Like it doesn't, your numbers don't like diverge on both sides. That's what LayerNorm kind of does. Does that kind of... Okay. Any other questions?

960. 1:49:44

     Okay. Um, so all we... Remember the decoder style... Oh, I think we actually kind of finished reviewing the Llama architecture. [laughs] There's nothing else to do. Um, the decoder, right, you do this 32 times.

961. 1:49:56

     Remember like four decoder layer is in self.layers. You do this 32 ti- I think it was 32, I can't remember, um, multiple times. Um, that's the decoder. You just do, you apply this multiple times.

962. 1:50:06

     You do a LayerNorm. And finally, you get your logits. Where is it? Your LLM head, right? This outputs the probabilities of which token. Remember, we're trying to predict the next token.

963. 1:50:19

     We output probabilities for each token, and that is called the LLM head. Um, and where's the forward function? The forward, right? There's a forward. Right, always with the forward.

964. 1:50:31

     Um, soft... You do, you go through the model, and then you... Okay, remember, ignore this, right? Ignore this, and use just LLM head. That's just [REDACTED:generic_id] line. [REDACTED:generic_id] line, okay?

965. 1:50:40

     [REDACTED:generic_id] line, and then you do the float. Now, another question people have is, like, why do you have to do the float? Um, um, does anyone know why you have to do...

966. 1:50:48

     You have to upcast a float. Why? Any clues? Have a guess. I'd rather make this bigger.

967. 1:50:57

     Have a guess. Have a guess. Have a guess. Why do we have to upcast a float?

968. 1:51:02

     Gradients.

969. 1:51:03

     Sorry, gradients? Okay, close. Why? Why gradients? It is related somehow to gradients.

970. 1:51:10

     Anyone care to guess? Okay, it's for training stability purposes. So the softmax, you should always upcast to float 32 because it makes training more stable. If you take the der- derivatives and the gradients, if you do not do float 32, you might get NaNs as well.

971. 1:51:27

     Remember, the exponential is, can be very large, so you want to take the float 32, which has larger precision than float 16. Right, float 16 is, the maximum number is 65536, I think.

972. 1:51:38

     I think it's 65536, right? But float 32, the maximum is, like, some large number to the power of 38 or something. 10 to the power of 38. So that's why you have to upcast it to, to float 32.

973. 1:51:49

     This just makes training more stable, right? So all of these things that we do tricks, it's just to make training stable. Yes.

974. 1:51:55

     You said you're doing the loop operation 32 times. Is that just, like, an ar- arbitrary thing that people figured out works, or is there a reason to put it-

975. 1:52:02

     Oh, that's up to you. So, like, if you wanna do more parameters, you can do 300 times. Up to you. That's just make your model 10 times larger. So, like, when you see, when you hear, like, you know, Llamas...

976. 1:52:12

     Yeah.

977. 1:52:12

     Each time you do it, it's generating, like, a weight for the model? Is that each layer, like, each time you iterate it's gonna generate, like, a, a set of weights?

978. 1:52:20

     Is that what it is? Or is it-

979. 1:52:20

     So, so the weights you train-

980. 1:52:22

     Yeah

981. 1:52:22

     ... when you take the tokens, you go through the architecture and, like, it changes the out- it changes the tokens.

982. 1:52:28

     Mm-hmm.

983. 1:52:28

     And it, these tokens keep shifting to, like, some sort of, like, new direction, and you keep doing this.

984. 1:52:32

     So but if you do it more times, you, do you get a, you said you get a larger model from, from that?

985. 1:52:37

     The problem is you have to train more weights.

986. 1:52:39

     Oh, that's so-

987. 1:52:40

     So each, each 32 times has different weights.

988. 1:52:43

     Right. Got it. Right, right. Okay. So each iteration has its own-

989. 1:52:46

     Correct. Each of the, each of the iteration... Well, it's not... Yeah. Each, there'll be 32 different weights for each layer.

990. 1:52:52

     Got it. Okay.

991. 1:52:53

     And so, like, yeah, normally people just tr- if you see, like, there's, like, you know, GPT-4, what is it? Like, [REDACTED:generic_id], [REDACTED:generic_id] something trillion tokens. I'm assuming there's more layers, larger embedding dimension, larger this, larger that, more layers.

992. 1:53:05

     Um, normally speaking, the more layers you do, the model can learn more. Um, so that's the whole reason why you wanted to add more layers. You just want to increase the capacity of the model to learn.

993. 1:53:14

     Um, again, is to make training more stable, again. Um,

994. 1:53:19

     and so this, remember the shifting trick that we did? In PyTorch, the shifting trick is just this and this. That's the shifting trick. That's the thing that makes it learn to the, predict the next token.

995. 1:53:31

     Um, and then you pass it through the loss function, the cross entropy loss which we discussed about, um, and then that's the Llama architecture, and that's no more. The rest is not useful.

996. 1:53:42

     The rest is... Yeah. So in theory, you could write the entire Llama architecture in, like, I think 50 lines or something. Um, the rest is just unnecessary bloat, right?

997. 1:53:53

     This, all of this is 1,600 lines of comments and stuff like that, but, you know, this is for Hugging Face's implementation. I s- It's highly respected. Um, and this is what you should look at first when you read a new architecture.

998. 1:54:04

     Um, so we just kind of went through the Llama architecture. Hopefully you can kind of get a sense of it. Obviously this is, if this is your first time reading the source code, um, it's actually not that hard.

999. 1:54:14

     It's not that complicated. You just have to see which components you can ignore, right? It's not that scary. Um, yep, does that kind of get it? Or do you guys kind of get that feel?

1000. 1:54:26

      Um, we're gonna do more, obviously. This is just the first [REDACTED:generic_id]. Um, any questions?

1001. 1:54:29

      Might be a dumb question.

1002. 1:54:32

      Yeah.

1003. 1:54:32

      Is there a major architectural difference between Llama 2 and Llama 3?

1004. 1:54:36

      Um, no, not really.

1005. 1:54:38

      No.

1006. 1:54:38

      Other than more tokens. Um, I think they changed embed... They did change some of the numbers, like how many, how many numbers you want to represent for each number, they changed that.

1007. 1:54:47

      Um, large vocabulary. They did much larger vocabulary and more tokens. Other than that, uh, no, there's no change at all. Yeah. Yes.

1008. 1:54:56

      Um, it looked like a lot of code you were writing was in Triton. Do you find, like, any, um, reason to ever drop into CUDA, or is, like, Triton pretty much, like, always the best?

1009. 1:55:07

      The reason why... It's funny, I used to work at NVIDIA. Why should I not be writing CUDA, right? The reason is, I see CUDA as extremely annoying to write, and if you want to optimize for just NVIDIA hardware, okay, go ahead, you can do CUDA.

1010. 1:55:21

      But my view is, like, I don't think so, that's going to be forever. So as a, as, like, a, um, as, like, a safety p- precaution, let's just do Triton, right?

1011. 1:55:32

      Let Triton handle the compile, like, compiling down to CUDA or AMD or whatever, Intel or whatever, right? And Triton can be the intermediary, intermediary. If you wanna get, like, 10% faster, yes, you should do CUDA.

1012. 1:55:43

      But it's only 10%, right? If you do fine-tuning [REDACTED:generic_id] times faster, it's already, like, it's already nearly at the ceiling. You can only go so much. Um, and so, like, if you wanna go down the extra mile, yes, more than happy to welcome you to do that.

1013. 1:55:54

      But I don't know. I do not l- It's funny because I used to do CUDA all the time, but I don't suggest it. Um, it... You will get more performance though, but I don't suggest it.

1014. 1:56:02

      Yes, question.

1015. 1:56:02

      So you never had to drop down to CUDA, right?

1016. 1:56:05

      Oh, sorry. Yes, what?

1017. 1:56:05

      You never had to drop down to CUDA. You could live with Triton.

1018. 1:56:08

      Yes, you don't... Yeah. So, so Triton, you write it in Triton, then it compiles down to CUDA. Yeah. Sorry. Wait. Yeah, yeah

1019. 1:56:21

      Actually, it could work. The only problem why it doesn't work on AMD is Triton. Oh, I think. And Xformers. Actually, I ... If ... So if Triton works on AMD, we work.

1020. 1:56:34

      If Triton, if Xformers, so Facebook's, um, flash attention library, if that works on AMD, then we work. Oh, funny, we work. But anyways, it depends on those conditions. So if, if AMD has those to work, then yes.

1021. 1:56:48

      I- in theory, you can remove Xformers and just use scale dot product attention, so there's only [REDACTED:generic_id] dependency, which is Triton. Um, I think some people have gotten it to work, so it depends.

1022. 1:56:57

      Yes?

1023. 1:56:58

      I kind of have an answer to that. I've trained, uh, on a, uh, MI300X Instinct, uh, with [REDACTED:generic_id] card with Triton, and it worked, um, with AMD. So-

1024. 1:57:09

      Okay. I mean, if Triton works, then yes

1025. 1:57:11

      ... it, it, it just works. So, uh, I just, I, I just have an answer, sorry.

1026. 1:57:16

      Okay, then, no, good. You answered, yeah. Okay. Yeah, but we don't ... So officially, we do not support AMD, um, but I guess it, it works. Okay, that's interesting.

1027. 1:57:25

      Um, yes. Okay. Um, what's next? Where's my ... Where's the Gemma [REDACTED:generic_id]? Uh, yes. Okay, so we're now gonna be talking about Gemma bugs, um, specifically Gemma. So if you go to our blog post, I actually ha- we wrote a blog post, um, about all the issues that we found in Gemma.

1028. 1:57:44

      Um, for example, you must add a BOS token. There is a typo in the paper. Um, yes, so we don't just find bugs and ... You know, we have to read the paper first to understand the model.

1029. 1:57:54

      Um, now, the problem is sometimes when people release models, they don't release papers. That is very painful. That happens a lot now. So please, model creators, please provide papers, um, otherwise it gets more complicated.

1030. 1:58:08

      Um, there's also, like, some other issues. Um, and we have a Colab notebook which provides all these ... Um, so if you open up the link, Gemma details, um, in the ...

1031. 1:58:19

      Remember, if you don't have access to these slides, it is tinyurl.com/unsloth, right? Um, that's the slides. Um, if you open up the Colab notebook, um, this is actually runnable in Colab.

1032. 1:58:33

      Please log into your Gma- uh, Google account for this to actually work. Um, but we show that this is the log L2 norm, so we check the ... So this layer number, right, there's like 18 layers.

1033. 1:58:45

      We check every single layer, the output of the actual imple- good implementation, so the DeepMind implementation, with the Hugging Face [REDACTED:generic_id], with the PyTorch [REDACTED:generic_id], with the other ones, and if you do the L2 norm, you find that the error is very high.

1034. 1:59:00

      And we, what we do, we showed is that you can actually move the error down by doing multiple changes, right? So each line, you can see there's, like, multiple lines.

1035. 1:59:07

      Each line is actually a method that we apply to make it better, right? So, like, we finally found that approximately either the blue line, or maybe it depends on which [REDACTED:generic_id] you like, either the blue line or the black line makes training much better.

1036. 1:59:21

      Um, does anyone notice any interesting things about that, this graph?

1037. 1:59:27

      Anything interesting? Like, do you see the ... You know, the ... So remember, each line is a fix that we did, right? So, like, there's many lines, and we did a fix, and it changes the, the error.

1038. 1:59:38

      What ... And we selected the black line to be the final [REDACTED:generic_id].

1039. 1:59:43

      Does anyone have any ... What is, like, anything interesting?

1040. 1:59:47

      It looks like the [REDACTED:generic_id] that did the most, like there was a huge jump than, than the rest of them.

1041. 1:59:52

      Yes. So [REDACTED:generic_id] of them caused a huge jump, and that is a RoPE float 32 fix that we did for all architectures, yes. And the other ones are less prominent.

1042. 2:00:01

      But anything else? Anything else interesting? Yes.

1043. 2:00:03

      Yeah, so the, the next line, you know, suddenly goes up and then it drops. Like, why?

1044. 2:00:08

      Yes. Fantastic. Why? I do not know. [laughs] And that is a good question, and I don't actually know. I think it's just language. You know, I have a theory. The theory is-

1045. 2:00:19

      We're here for you to tell us

1046. 2:00:20

      ... yeah, but unfortunately, I can't say everything. Like, I, I mean, my theory is ... And there was also a jump as well in the middle. Um, and the blue line, you know, it starts from very low, it goes up very high, and everything does this, right?

1047. 2:00:31

      So, like, there is just some weird transition boundary, um, in the Gemma model, right? And so, like, I'm just gonna guess. Um, my guess is that when you do it, when you train a transformer, the layer, the later layers get harder and harder to train, right?

1048. 2:00:46

      The earlier layers actually get very easy to train. And so this transition boundary is when the model probably was not really trained that well. Um, so I'm going to guess, this is just guessing, um, that maybe the model should have been trained for more data, um, and the boundary should disappear.

1049. 2:01:01

      Um, this is just my guess. Um, so there is a phenomenon essentially is, like, more data- the model, the la- l- the la- the last layers are much harder to train, and that's kind of my theory, but I don't, I don't, I don't think that that's correct, but ...

1050. 2:01:17

      Thanks.

1051. 2:01:17

      Okay.

1052. 2:01:17

      And the blue [REDACTED:generic_id] kind of drops for a moment, right?

1053. 2:01:22

      Yes.

1054. 2:01:23

      Right before the, the last [REDACTED:generic_id].

1055. 2:01:24

      Yeah, exactly. Um, so in the end ... So now the question is, like, why did we choose the black [REDACTED:generic_id] then? Why didn't we choose the green, the blue line?

1056. 2:01:32

      Um, so that's adding the exact gelu that we found. So if you add the RoPE fix plus the exact gelu, you get the blue line. But we, in the end, decided to do the bl- black line, and why do you think that is?

1057. 2:01:43

      We did not choose the blue line. We should have chose the blue line, right? But the, uh, with final, the all the fixes that we did ... So essentially, the answer why we did not choose the blue line, the blue line should actually have ha- lower error, right?

1058. 2:01:53

      The reason why we didn't choose that is because there was not just [REDACTED:generic_id] error, there were [REDACTED:generic_id] errors. There was many errors, and all of the errors compounded together, we finally chose the black line because it matches the original implementation.

1059. 2:02:06

      So because remember, the trick is you have to match the original implementation of whatever the Gemma models created, the Gemma model creators did. Um, so you can't, like, just look for this error.

1060. 2:02:16

      Maybe, like ... I, I mean, maybe, like, if someone ch- ch- chose different, like, fixes that we did, you can probably get even a lower training loss. Um- I guess you could.

1061. 2:02:27

      Um, but we decided to choose the black line because that's what the original implementation did. Um, any other questions?

1062. 2:02:36

      For, for my own clarity here. So I know the weights, they're trained, and, uh,

1063. 2:02:45

      when you, when you kept saying the model could have been trained, you're not talking about the weights, were you?

1064. 2:02:51

      Oh, I am talking about the weights.

1065. 2:02:52

      Okay.

1066. 2:02:52

      So the weights are the ones in the... So the model weights are the ones training, right? So the rest you don't actually train, it's just the weights itself.

1067. 2:02:59

      Yeah. Yeah, 'cause it's a bit meta, right? Like, at the end, you end up with a model that's a transformer, and then the weights themselves had to be trained.

1068. 2:03:07

      So anyway, that leads me to my follow-up question, which is,

1069. 2:03:11

      um, do you have examples of that weight training? It... Like, is it...

1070. 2:03:19

      Like, I guess, what, what are those training based off of? And is, is it iterative where you're just gonna see the training loss over time, and you're just trying to find the, the right weights that give you the less, the least amount of loss?

1071. 2:03:34

      Yes. So remember the, the, the goal of a transformer is you want to predict the next word, right? So the sentence, "Hello, my name is Daniel," you're trying to predict hello, predict my, right?

1072. 2:03:43

      My, predict name, and so on. You have this data, correct? Like, you have... Just take novels, you shove in the novels, you already-- You essentially created data out of thin air, and then you change these weights using, like, backpropagation, do derivatives, and try to, like, change these weights such that you get the highest accuracy.

1073. 2:04:03

      Um, and this training procedure is called backpropagation. Um, and so, like, I was trying to show you, like, how do we actually derive the derivatives. When you do backpropagation, you need to derive the derivatives.

1074. 2:04:12

      Um, just use PyTorch. PyTorch will do the derivatives for you. Um, and yes, but that's... Does that kind of answer your question or...?

1075. 2:04:19

      Yeah, I think that gives me a glimpse of, of some of that stuff, yeah.

1076. 2:04:22

      Okay. Yes.

1077. 2:04:23

      So you mentioned the middle layers are, you know, layer layers are usually under-trained.

1078. 2:04:27

      Yes.

1079. 2:04:28

      And often for fine-tuning, you see that the first layers get over-trained. I know that people are doing layer freezing now, but is there a way to change the learning rate dynamically based on the layer?

1080. 2:04:37

      Yes, so Unsloth actually has that. [laughs] So you can actually, depending on your layer... So for now, what we do is your embedding and your final layer, you can change different weights, uh, different lay- learning rates.

1081. 2:04:47

      So we found that if you train on, if you train the last layer with the embedding weights and the first... Sorry, the embedding weights in the LLM head by a factor of 10 smaller, the learning rate, you can actually have increased accuracy.

1082. 2:04:59

      So you actually... So yes, you should. You should change the lay- you should change the learning rates for each layer. Um, but people don't actually do that. I think it's because if you set a learning rate for each layer beforehand, you're kind of like,

1083. 2:05:13

      it's like there's, you're, like, doing subjective bias. So, like, that's why people just sort of set [REDACTED:generic_id] learning rate for all the layers. Um, but I think in this case, I'm just gonna guess, okay, this might be a transformer.

1084. 2:05:22

      This is transformer general. This is not just for Gemma, this is for all transformers. Maybe, I guess, lay- layer-wise learning rate could work. I think there are, like, some papers which do that.

1085. 2:05:32

      I think it's called LARS. I think LARS does layer-wise, um... I think it's called LARS, layer-wise, um, learning rate. Um, if hope- hopefully that answers your question. Um, y- yes.

1086. 2:05:44

      Yeah, where does what the Y axis represents?

1087. 2:05:47

      Oh, the Y... Oh, it's a log L2 n- norm. So it's, you take the DeepMind implementation, you code it up correctly, then you take the other implementations like PyTorch, Hugging Face, even DeepMind's own implementations, and then you check each layer, the output, you compare it with the original imple- the correct implementation, and check what's the error, and

1088. 2:06:08

      that's the thing that I graphed. Um, and your goal is you want the error to go about, go to [REDACTED:generic_id], right? So, like, you want it to go all the way to [REDACTED:generic_id], so, like, you know, on the bottom, and not, not, like, very high.

1089. 2:06:16

      Um, and that's log scale, right? So the error is not, like, a small number, it's 1,000, right? So, like, every single line, every single step you go down is a log difference, right?

1090. 2:06:26

      It's not, it's not like a s- It's, I essentially logged it. If you did not log it, it would look very bad. Um, but I just logged it. Um, yeah.

1091. 2:06:33

      Does that... Okay. Any other questions? Yeah.

1092. 2:06:36

      Uh, so let's say if there's an issue in a tokenization, uh, part-

1093. 2:06:41

      Uh-oh

1094. 2:06:41

      ... a fundamental-

1095. 2:06:43

      Mm-hmm

1096. 2:06:43

      ... thing, or we find some optimization-

1097. 2:06:45

      Mm-hmm

1098. 2:06:46

      ... uh, or, or a bug, and you have to change the way you're tokenizing-

1099. 2:06:49

      Mm-hmm

1100. 2:06:50

      ... uh, your sentences and everything, would you have to retrain your models to, like, incorporate these fixes and changes to a tokenizer?

1101. 2:06:58

      This actually happens a lot, very frequently, and, uh, I think, like... So, like, for example, TinyLlama, someone trying Ti- TinyLlama, and then training already 80% completed, they found a bug for tokenization.

1102. 2:07:08

      They're like... So it happens very frequently, and it depends on what you wanna do. I think it depends to the model creator. If you already spent millions of dollars, maybe just keep, just train it with the bug, and then you release the bugged version, but it should still work, um, hopefully.

1103. 2:07:26

      Um, yeah.

1104. 2:07:27

      So in theory, let's say if OpenAI would have a lot of difficulty shifting if they found, like, if somebody else found a, a more optimized tokenizer or something like that-

1105. 2:07:37

      Mm-hmm

1106. 2:07:37

      ... they would have trouble shifting to that model because they would have to spend, like-

1107. 2:07:41

      You have to retrain everything

1108. 2:07:42

      ... millions of dollars.

1109. 2:07:42

      Correct. So you just, just assume it, just leave it. If you already spent, like, billions of dollars, [laughs] probably not a good idea to retrain. Um-

1110. 2:07:50

      So they have-- So even if it offers, like, a, like, 2X optimization or somehow, like, they would have to spend, like, they would have to retrain and spend-

1111. 2:07:57

      Yes, you have to retrain everything from scratch. Um, but that's why, like, I think, like, that's why you should do, like, small scale experiments. You know, get, like, a smaller model, train it for less data, test it, and then see if the accuracy is good, and then you scale it up.

1112. 2:08:09

      Um, yeah. Any other questions? Okay. I will... Yes. So there's a notebook. So we show step by step exactly what we did, and if you inspect the code... Okay, the Gemma code is now, the Gemma code, um,

1113. 2:08:27

      if you... Oh, okay. Wait, no, it's modeling Gemma

1114. 2:08:34

      Oh, okay, maybe I should just go to Hugging Face itself. Um, wait, let me go to... You can actually find this, um, if you copy paste this,

1115. 2:08:42

      right? You edit the-- you go to Gemma

1116. 2:08:46

      and you go to modeling Gemma. Oh. Right, this is... Oh, did I not... Okay, let me just... Okay, maybe I typed it wrong. Did they not... Oh, okay, maybe I did [REDACTED:generic_id] Ls, my bad. [laughs]

1117. 2:09:00

      I always get confused on that. Um, oh, what is this? Hmm, this is interesting. Okay, this is, like, new. Um,

1118. 2:09:09

      so, okay, yeah, I did not... Yeah. So all of this, so we wrote inside the f- like, you know, Llama does... So we showed, for example, in the code now, if you go to, like, Hugging Face's code for Gemma, we wrote-- I tried to write some comments, you know, for it to be more clear on why we

1119. 2:09:25

      are doing this. Um, and so for example, the, uh, the, the layer norm, right, you have to be careful to where you up cast and down cast. Um, and we write this in here.

1120. 2:09:36

      Um, where is it? I think it's... No, no, no, not. Oh wait, is it... No, I'm pretty sure I've read it somewhere. No, it is here. Yes. Okay, it's a bit unclear.

1121. 2:09:45

      I need to, um, make this bigger. Okay, it's a bit blurry. But you can see that depending on the model, um, in Gemma, you have to actually up cast to float32 everywhere.

1122. 2:09:56

      You must use float32 everywhere because that's the original implementation used float32, right? So you must always follow the original implementation. If you don't follow the original implementation, then you will get wrong, like, you know, somewhat worse results.

1123. 2:10:08

      Um, and the problem was other implementations just copied Llama and Mistral's code, and they did not do this. And so we found that you actually have to up cast correctly over here, right?

1124. 2:10:21

      You have to up cast immediately. Um, and then you down cast at the very end. Um, and so we wrote a few comments, right? Llama does x.to float16, whilst Gemma is x, you know, it really, like, Llama does that, right?

1125. 2:10:35

      But Gemma does this, right? So there are, like, small little issues, uh, down casting, up casting. Another question is, like, why do we have to do down casting? Does anyone know why?

1126. 2:10:44

      Like, why is there always, like, down casting, up casting? float32, float16, float8. Does anyone know why we have to do down casting, up casting? Performance gains, but, uh, lower precision.

1127. 2:10:55

      Yes, correct. It's for faster speed. Um, so do you know how much faster? Like- Eight? Uh, no. So float32 to float16. What do you think?

1128. 2:11:07

      It depends what you do with it, right? It depends.

1129. 2:11:12

      [REDACTED:generic_id]. T- who said [REDACTED:generic_id]? Me. Okay, good guess. Why, why did you guess [REDACTED:generic_id]? Because that is the formula. [laughs] Well, that's for sparsity. Okay. Okay. Yes. Okay. float8, approximately [REDACTED:generic_id].

1130. 2:11:25

      Um, actually could be more. So float32 to float16 is actually not [REDACTED:generic_id]. It's actually much more. I think it's five, I think. Or is it six? Um, the reason is because the representation of the float is different, right?

1131. 2:11:38

      So float32, I have float, floating point representation Wikipedia. I think it's in here somewhere. Um, oh, maybe I go to bfloat16. Um, where is bfloat16?

1132. 2:11:54

      Brain float. B float? B float. Yes. Right. So, like,

1133. 2:12:01

      there it is. Um, oh, there's more pictures now. Oh, they edited this. I did not... Okay, this is new. I didn't see AMD FT24 format or Pixar. Oh, okay.

1134. 2:12:11

      They have, like, weird formats now. Um, this is float32, right? And float32, um, the exponent has eight numbers, right, eight bits, and the fraction of bit has twenty-three. Um, and when you do matrix, when you do matrix multiplication, does anyone know how to calculate the number of transistors you need for floa- float32?

1135. 2:12:34

      Does anyone know? It's a formula that's related to the exponent, the fraction, and the s- just the exponent and fraction. What do you think the formula is? Have a guess.

1136. 2:12:42

      Right. I said that it's approximately... So if you have bfloat16, the fraction is seven. Right. bfloat16 has sixteen bits you can use. The exponent, the exponent is used for the li- the dynamic range of the number, right?

1137. 2:12:56

      So the... If you want larger numbers, you have to have larger exponents, right? So this means bfloat16 o-only has a range of [REDACTED:generic_id] to the pow-- Okay, it's not [REDACTED:generic_id] to the eight, but, like, just assume, you know, it's [REDACTED:generic_id] to the power of eight.

1138. 2:13:07

      Okay, that's not right, but just assume that. Um, [REDACTED:generic_id] to the power of eight, right? But

1139. 2:13:13

      it's, yeah. Um, and this [REDACTED:generic_id], float32, also has [REDACTED:generic_id] to the power of eight. There is another format called float16, which is [REDACTED:generic_id] to the power of five. Um, and then the fractional component is ten.

1140. 2:13:24

      So all of these numbers you can scale, right? How many do you want for the exponent? How many do you want for the fraction? You must include the signed bit.

1141. 2:13:30

      And the trick is you must have sixteen. You need to fit, you know, sixteen. So you could have, like, exponent [REDACTED:generic_id], and fraction could be, um, fourteen. That could also work.

1142. 2:13:41

      Um, but does anyone know how many transistors do you need to use for float16, for example?

1143. 2:13:47

      And bfloat16? Remember I said it's, like, around five times faster. Well, it's actually not right. I think it's even more. Um, what is the formula? Have a guess. How many transistors do you need to use to do float16 multiplication approximately, or float multiplication?

1144. 2:14:01

      It's, it's a formula related to exponent and fraction.

1145. 2:14:07

      The answer is exponent plus fraction squared. That's the answer. Um,

1146. 2:14:13

      so what does that mean? That means float16 is five plus ten squared, right? And float32 is eight plus thir- twenty-three squared. So it is not [REDACTED:generic_id] times faster. It is much faster, right?

1147. 2:14:28

      So, like, I don't know what that is. What is, um...

1148. 2:14:31

      So it's eight plus twenty-three squared Five hundred and... So you need a, approximately, okay, this is approximately, you need 537 transistors for fl- float 32 multiplication.

1149. 2:14:45

      And then that whole 23 squared or just 23?

1150. 2:14:47

      Oh, it's just 23 squared. So it's eight plus 23 squared.

1151. 2:14:50

      Oh, right.

1152. 2:14:50

      Yeah. And so what, what is the other [REDACTED:generic_id]? I think it was, um, was that... I can't remember.

1153. 2:14:56

      Five-

1154. 2:14:56

      So, um, so it's eight and seven, right? So eight and seven. This is, this is Google's format. It is 57. So what does that mean? How many times faster?

1155. 2:15:06

      10 times faster.

1156. 2:15:06

      Yeah, so it's actually 10 times faster, right? So float 32 to float 16-- bfloat16 is around 10 times faster, right? Float 16 is five plus 10, right? So five plus 10.

1157. 2:15:18

      So bfloat16 is approximately [REDACTED:generic_id] times faster than float 16, although in terms of no [REDACTED:generic_id] really notices any difference. Um, but in general, bfloat16 is actually faster, right? So that's why it's not [REDACTED:generic_id] times faster, it's 10 times faster.

1158. 2:15:32

      Um, and that's why you must use Tesla T4s, as I said, because it has Tensor Cores which does this, right? The Tensor Cores does float 16 multiplication very effectively and very efficiently.

1159. 2:15:41

      Um, and so do not use P100s again, right? P100s do not have this methodology. Um, yes, question.

1160. 2:15:49

      Why don't we go even smaller than 16-bit?

1161. 2:15:52

      Yes. Float eight. So float eight, I don't know, um, there are [REDACTED:generic_id] formats for float eight.

1162. 2:16:00

      Uh, oh, wait, I don't think so it's in Wikipedia. Float eight.

1163. 2:16:05

      Oh, okay.

1164. 2:16:06

      Yeah.

1165. 2:16:07

      Uh, floating-

1166. 2:16:09

      Yeah

1167. 2:16:09

      ... point. There is, it's called EM... Oh, it's, uh, just use min Floats. Does it... They have some. Yeah, there we go. Right. So you get to deci- remember, if you want to have eight bits, you get to decide how many you want to do for the exponent, how many you want to do for the fraction or

1168. 2:16:26

      the mantissa part, right? You get to decide, and depending on your-- the company, you know, it, it, it's, it's unclear. There's no standard. Um, so this [REDACTED:generic_id]'s [REDACTED:generic_id], four, three, right?

1169. 2:16:36

      So, like, what's [REDACTED:generic_id], four, three? [REDACTED:generic_id] plus four, right? Four, three squared, right? Is that, is that four, three? Yeah. 13. So float eight is...

1170. 2:16:47

      Four.

1171. 2:16:47

      I think it's around, yeah, so around four times faster than bfloat16. But in general, it's not. Okay? In general, it's like [REDACTED:generic_id] to three. It's, it's not gonna be four.

1172. 2:16:55

      The reason is because you're packing so many transistors in, you also have to do, like, energy. You have to, like, do the data movement. There's, like, other transistors you have to do.

1173. 2:17:02

      I just-- approximately it's [REDACTED:generic_id], [REDACTED:generic_id] to three times faster. That's float eight. Can you go even lower?

1174. 2:17:09

      Yes. Why don't we go [REDACTED:generic_id]?

1175. 2:17:12

      [REDACTED:generic_id] bit.

1176. 2:17:14

      [REDACTED:generic_id] bit.

1177. 2:17:14

      So you must have the sign, though, so you can't do [REDACTED:generic_id] bit.

1178. 2:17:18

      So 1.58 bit some people have been talking about. Um, [REDACTED:generic_id] bit. [REDACTED:generic_id] bit could be possible. The problem with [REDACTED:generic_id] bit is

1179. 2:17:26

      it's problematic because when you do [REDACTED:generic_id] bit training, um... Yes. Okay, so let's see. Let's, let's do [REDACTED:generic_id] bit, right? So what do you wanna do? How many exponent?

1180. 2:17:34

      [REDACTED:generic_id]? And remember, you have to have a sign bit. That's the most important.

1181. 2:17:38

      [REDACTED:generic_id] maybe.

1182. 2:17:39

      [REDACTED:generic_id] for the exponent and fraction [REDACTED:generic_id], right? That's because remember it's squared, so plus [REDACTED:generic_id]. Oh, wait, no, it's, it's [REDACTED:generic_id] plus [REDACTED:generic_id]. Okay, so it's [REDACTED:generic_id]. Okay.

1183. 2:17:51

      Uh, mm, 10 times faster? I don't think so. Um, okay, maybe, maybe [REDACTED:generic_id] bit is probably too low. Maybe four bit. Four bit could work. Yeah. Yes?

1184. 2:17:59

      I have a question because, you know, this math fit the flops on, say, NVIDIA's spec sheet because I understand this math, but they, for example, have the same, uh, tensor flops for BF16 and FP16.

1185. 2:18:13

      They don't do it-

1186. 2:18:13

      Oh, that's just because they wanted to do that, just for easier calculation. Like-

1187. 2:18:18

      And then for their Tensor32, they do list it 2F, so that's why I would guess-

1188. 2:18:22

      Tensor32 is not 32.

1189. 2:18:24

      Okay.

1190. 2:18:24

      It is... They have it somewhere. NVIDIA. Tensor float, it's 19.

1191. 2:18:31

      That's the trick. They like to, like, do marketing and they say it's 32, but it's actually 19. Um, yes, that's why it's the same. Okay, any other question? Was it someone else raised their hand or...

1192. 2:18:42

      Okay. But yes, I was gonna say, like, you can do four bit, right? So four bit is actually a new-- NVIDIA's new GPUs, the, um, the B100s do have flo- four bit.

1193. 2:18:51

      Um, so that is approximately [REDACTED:generic_id] times faster. Now, the reason it's not... Okay, let me just try. Four bit. I think it's [REDACTED:generic_id] plus... It's, it's probably, like, [REDACTED:generic_id] plus [REDACTED:generic_id] or something.

1194. 2:19:02

      I don't know. Six. Okay. Right. Um, it's not gonna be that much faster because, as I said, there's power transistors and there's other transistors. You can only go so far.

1195. 2:19:11

      Just the jump from float 32 to float 16 was very large. Um, yes. Yes.

1196. 2:19:15

      So quick question. Um, so for the example, the [REDACTED:generic_id] bit BitNet, that-

1197. 2:19:20

      1.58 bit? Yeah.

1198. 2:19:21

      Yeah. So that would be an example of [REDACTED:generic_id] bit, right? In this.

1199. 2:19:25

      So it's a different-- So actually, I had a tweet about this. 1.58 bit and flo- float4 is the same in terms of number of transistors. You'd rather use float4.

1200. 2:19:33

      Okay.

1201. 2:19:33

      The reason why is 1.58 bit is you have to do more manipulation to make it work. You have to use, like, the straight through estimator, like the, uh... It's a, it's a horrible mess.

1202. 2:19:42

      You'd rather just use float4. Um, so float4 and 1.58 bit are, like, similar. Um-

1203. 2:19:47

      You get to create your own base model if you do a, a [REDACTED:generic_id]... If you rep- if you replicate it, if you replicate the paper.

1204. 2:19:55

      Yes. What, what do you mean?

1205. 2:19:56

      So which most of us have never done, right?

1206. 2:19:57

      Mm-hmm.

1207. 2:19:58

      Which would be-

1208. 2:19:59

      I, I mean, Technium and the Noah's research probably replicated their sum, though. It does work somewhat. Somewhat works. Um-

1209. 2:20:07

      I mean, yeah, it's [REDACTED:generic_id] bit, but I mean-

1210. 2:20:08

      1.58, yeah. It's not actually [REDACTED:generic_id] bit. Yeah.

1211. 2:20:10

      Yeah.

1212. 2:20:11

      It's, I think it's like three.

1213. 2:20:12

      They call it [REDACTED:generic_id] bit, but yeah.

1214. 2:20:12

      Oh, yeah, they like to call it [REDACTED:generic_id] bit. Yeah.

1215. 2:20:14

      So, um, but my question is like, so in theory, like, obviously mo- uh, I don't know who works here, but like most of us have never built a base model, right?

1216. 2:20:26

      Yes.

1217. 2:20:27

      So like-

1218. 2:20:27

      Well, you could. Yeah. Yeah.

1219. 2:20:28

      Yeah, you can with enough GPU power, but BP-- but [REDACTED:generic_id] bit BitNet, you know, that was... And they even had like a really good great tutorial and like... So but do you think that that's just like-

1220. 2:20:40

      I'm just asking for your opinion on that

1221. 2:20:42

      I, I don't think so 1.8-- 1.5A bit will be the future. I think NVIDIA is, like, the focus is on float four. They might go to float... I think float four might be the final precision.

1222. 2:20:51

      I don't think you can go any faster with that. I think float four is the final, no more. So we won't be having that much faster GPUs. I don't think so.

1223. 2:20:59

      Um, I mean, float four is actually... They don't actually do float four anymore. It's actually float six for the gradients. Float six and then float four for the activations, like, you know, it's very weird.

1224. 2:21:07

      I mean, you could do, like, float three, float [REDACTED:generic_id], but, like, it's, you're diminishing returns. Um-

1225. 2:21:13

      So, um, in Arm silicon though, uh, there's, um, there's been, like, advances in like, uh, like super low bit width bits versus, uh-

1226. 2:21:24

      Fixed point stuff?

1227. 2:21:26

      Huh?

1228. 2:21:26

      Is it called fixed point stuff or-

1229. 2:21:27

      Huh?

1230. 2:21:28

      I, I think it's called fixed... I, I know Arm has fixed point. Um-

1231. 2:21:31

      Oh, well, yeah. So it's, uh... I mean, just like the Snapdragon X, like the new-

1232. 2:21:36

      Yes

1233. 2:21:37

      ... um, phone

1234. 2:21:37

      They have... So it's like customizable as well? Or I, I don't know.

1235. 2:21:40

      Yeah. Well, the... Okay, so the SDK is broken. You have to pass the, um... So this is why you can technically run Mixtral 8x7B on your phone at like 20-something FPS.

1236. 2:21:53

      Not FPS, sorry, TPS. Is because you can use UFS 4.0 as flash storage and subsequently use that as memory and then the, the... But the thing is then you're running at [REDACTED:generic_id]-bit precision, which is, um-

1237. 2:22:10

      So the, uh, that's probably why this... So if you use [REDACTED:generic_id]-bit precision, that's why you have memory reductions. But there is actually papers which show that if you do [REDACTED:generic_id] bits for the MLP plus four bits for attention, that's actually the most appropriate.

1238. 2:22:20

      You can actually do that.

1239. 2:22:21

      So that, so that's not a, uh, invalid approach.

1240. 2:22:24

      No, that's not invalid. That actually, it works. It works. The, the-

1241. 2:22:27

      I've seen it work

1242. 2:22:27

      ... Möbius people did that, I think. Yeah.

1243. 2:22:29

      That's all.

1244. 2:22:30

      Yes, question. Sorry. Okay.

1245. 2:22:33

      Uh, [laughs] [REDACTED:generic_id] kind of related questions on precision. First [REDACTED:generic_id] is, like, why is the negative bit, like you must have negative-

1246. 2:22:40

      The signed bit?

1247. 2:22:41

      Yeah.

1248. 2:22:41

      You don't have to, but it's generally, generally, like, standard practice to have the signed bit. In theory, you don't have to. Um, the only problem is if you don't have a signed bit,

1249. 2:22:53

      your numbers will be [REDACTED:generic_id], [REDACTED:generic_id], [REDACTED:generic_id], right? But the... what happens if you wanted to, like, make the model... Like you're, you're trying to not make the model learn negative directions anymore.

1250. 2:23:02

      You could do that. I don't know if there are papers which... Maybe you should write a paper about that. Um, train a model on that and let's see what...

1251. 2:23:08

      Okay. But yeah.

1252. 2:23:09

      Well, and very, and very related, I think this is all bits. Is, uh, it's all bits, right?

1253. 2:23:13

      Yeah.

1254. 2:23:14

      Uh, uh, softmax, you're basically just linearly, like, fitting stuff down to a certain number of bits. There's nothing special about softmax and things that could be like exponentially big sort of going this way, right?

1255. 2:23:25

      Like-

1256. 2:23:26

      The reason-

1257. 2:23:26

      Why this would be better

1258. 2:23:28

      ... it's because remember, when you do softmax, you also have to normalize by the sum of the exponentials. And if you do exponential of 10, you already get like some large number, and this, this probability will take over the entire sum.

1259. 2:23:39

      Well, but you're not, you're not, like, logging it. You're just, uh, square rooting it.

1260. 2:23:44

      No, no. It's, it's, it's the sum of ex- exponentials divided by... Oh, sorry, the exponential divided by the sum of the exponentials.

1261. 2:23:51

      Yeah, but like the biggest exponential dominates the sum, right?

1262. 2:23:54

      Yes. That's the problem, though. If you do that, then your model's not learning. You're just trying to learn to predict [REDACTED:generic_id] token. Why don't you just predict that [REDACTED:generic_id] token then?

1263. 2:24:02

      Like the, the largest [REDACTED:generic_id] that you did. That's kind of what you're... You're forcing the model to not learn anything. That is why you have to like minus the maximum.

1264. 2:24:10

      That, that's the trick that we showed, is like minus the maximum, and then you can like reduce this effect of this [REDACTED:generic_id] token, um, or this [REDACTED:generic_id] issue. Um, so it's for training stability purposes.

1265. 2:24:19

      Um, I don't know if that kind of... Okay, probably that didn't answer your question, but okay. Yes.

1266. 2:24:24

      From a practical perspective, how much slower was upcasting the LayerNorm? And then do we know whether that's actually more accurate than the way that, uh, Llama does it?

1267. 2:24:35

      That is a good question. To be honest, I do not know. I don't think so it changes too much. Um, LayerNorms, if you upcast, um, it's probably, yeah, small effect.

1268. 2:24:46

      Small effect. But the reason why you need to upcast is because Gemma did it before, so you have to do it. Um, remember the trick is you must follow what the original implementation does.

1269. 2:24:55

      Um, any other questions or... Okay. There are like some other issues which we showed, um, that more... Okay, this-- It's funny because it's all about upcasting, downcasting, and stuff like that.

1270. 2:25:05

      Each implementation had-- does its own thing. Um, unfortunately, how do you actually analyze this? You have to open three screens up. The DeepMind [REDACTED:generic_id], the, ooh, z- the DeepMind [REDACTED:generic_id].

1271. 2:25:16

      Okay. [laughs] Okay, I'm too excited. Um, you have to open up three implementations, DeepMind [REDACTED:generic_id], the Hugging Face [REDACTED:generic_id], the Keras [REDACTED:generic_id]. You have to open up three screens, and you see line by line what did they do, and then now you have to guess which [REDACTED:generic_id]'s the correct [REDACTED:generic_id].

1272. 2:25:29

      Um, the guessing part is the most painful. So you have to like inquire. You ask Hugging Face which [REDACTED:generic_id]'s the correct [REDACTED:generic_id]. You look at the paper, which [REDACTED:generic_id]'s the correct [REDACTED:generic_id].

1273. 2:25:38

      You assume the DeepMind [REDACTED:generic_id]'s correct and stuff like that. So there's like some human component, you have to guess. Um, guessing. So that's probably why it can't be automated, right?

1274. 2:25:46

      These error checking things cannot be automated is because the... there's a human there which made these, made these decisions. And so you have to, now you have to decide which [REDACTED:generic_id], which of those decisions did they choose.

1275. 2:25:57

      Um, and you can't really automate this way. I guess you could automate this by doing the methodology which we described. Try all combinations and see which [REDACTED:generic_id] has a lower error.

1276. 2:26:06

      I guess you could do that. But remember, you must have the original implementation first. Um, that is a problem. Um, so there's like chicken and egg problems. Um, the RoPE position, um, so this is the [REDACTED:generic_id] I was talking about, upcasting RoPE.

1277. 2:26:19

      Um, this is in all architectures now. You must not downcast RoPE. If you do, you'll get wrong results. So previously on the left, right, if you see [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id], [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id], [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id] [REDACTED:generic_id], that's the positions.

1278. 2:26:36

      Um, that is definitely incorrect. Right? What does that mean? Like, do you know why that's incorrect, eight [REDACTED:generic_id] nine [REDACTED:generic_id], eight [REDACTED:generic_id] nine [REDACTED:generic_id], eight [REDACTED:generic_id] nine [REDACTED:generic_id]? Does anyone know why?

1279. 2:26:43

      Remember, this is positions. Why is it-- Why is it all the same? Like, wh- does anyone know why this is very bad?

1280. 2:26:50

      So we kind of like ... Essentially now we ... The three words have the same position, right? 8192 is the position. And what is another big error of this?

1281. 2:27:00

      There's actually [REDACTED:generic_id] more error. Let's assume the maximum is 8192, the sequence length. What is 8192?

1282. 2:27:10

      The end or absolute the end.

1283. 2:27:13

      It's out of bounds. Remember, it's minus [REDACTED:generic_id] for Python, right? It's 8191 is the correct number, right? So if you correct this, you get 819- 8189, 8190, and 8191, right?

1284. 2:27:25

      And you can see all the numbers are like this. So the point is, if you use ... Remember, the whole point of this problem is because we're using float 16 for faster training.

1285. 2:27:33

      Remember, float 16 is how much times faster?

1286. 2:27:36

      Yes, around 10, or five to 10, something around there, right? That is why you have to do this, and these are the issues pop out because of this issue, right?

1287. 2:27:43

      We're trying to make training faster, but then these issues come. Um,

1288. 2:27:49

      and the Gelu [REDACTED:generic_id], which we described before, um, this was the first bug that we found. Um, actually I think this is the main reason why we were trying to look for bugs is we found that, oh look, there is this bug in Gelu, um, in the activation function.

1289. 2:28:04

      And so the point is Keras used approximate Gelu, the PyTorch and the PyTorch version used exact Gelu, and Hugging Face also used exact Gelu. Um, and the question is which [REDACTED:generic_id] is the correct [REDACTED:generic_id]?

1290. 2:28:17

      Is the exact Gelu correct? Is approximate Gelu correct? So what's the difference between exact and Gelu, um, Gelu activation function? Um,

1291. 2:28:27

      there is the ... Where is the ... I don't know if they have the exact and the, um ... It's called flax.

1292. 2:28:36

      Oh, okay, that's night mode. The ... Oh, that's even worse. Okay, whatever. Oh, wait, that's Prelude. Where is Gelu?

1293. 2:28:44

      Oh, wait, no, I have to find the right ... Oh.

1294. 2:28:49

      Yes. Right, so, like, the u- the exact Gelu is this [REDACTED:generic_id], right? There's a error function. Um, okay, my thing is not rendering it properly. Um, but if you ...

1295. 2:29:00

      Essentially what you do is you use Desmos. So what I like to do is I use Desmos, um, Desmos, right, and literally plot them toget- plot them on the graph, right?

1296. 2:29:09

      So, like, if you have, right, X, Y is equal to X over 2, right? You literally type this in, and what is this? [REDACTED:generic_id] ... I think you can do error function.

1297. 2:29:18

      Oh, yes you can, right. You can do error function, right, X divided by square root of [REDACTED:generic_id]. All right. That's the exact Gelu. Right, now you type in this complicated formula for X divided by [REDACTED:generic_id], um, [REDACTED:generic_id], [REDACTED:generic_id] plus tan what?

1298. 2:29:37

      I don't remember this. Square root of [REDACTED:generic_id] divided by what? Was it pi? Oh, it's pi. Pi and the what? Um, X plus [REDACTED:generic_id] point-

1299. 2:29:50

      4447.

1300. 2:29:50

      Can I ... Oh, we can't. Okay. 0447. Is it 0447. What was the

1301. 2:29:57

      ... Oh, [REDACTED:generic_id] five, [REDACTED:generic_id] five X to the cubed. Was it cubed?

1302. 2:30:01

      Yeah.

1303. 2:30:02

      Yeah. Okay, right?

1304. 2:30:04

      I mean, it's only the [REDACTED:generic_id] that is actually root [REDACTED:generic_id].

1305. 2:30:06

      Oh, is it? Oh, you're right. Okay. Wait, is it, is it ... Oh, is it just the rendering problem, or is it square root? No, no, no. It's square root of [REDACTED:generic_id] over pi I think.

1306. 2:30:15

      Wait, is it correct? Wait, something ... I did something wrong. Maybe I did something wrong. Oh, whatever. Who cares? Just, just assume ... Okay. Oh, wait. Oh, yeah, yeah, you're right.

1307. 2:30:23

      You're ... I, I put the square root everywhere. Oh, is that what you were saying? Yeah. Oh, okay. Oh, no, no. No, no. What? No. [laughs] Get rid of that.

1308. 2:30:33

      Okay, let me just ... No, it's, it's tan of everything. Now I have to do this.

1309. 2:30:38

      Oh. Yeah, yeah. This probably have to play around with this. Oh, there we go. There we go. Right. So the blue line, if you remove it, the blue line and the red line, right?

1310. 2:30:46

      They're the same thing. But what's the difference? Remember in Desmos, I don't know if people know this, we can actually do derivatives, D over DX. Did anyone know this?

1311. 2:30:57

      You can actually do derivatives. You get your D over DX, and then you can do this as well,

1312. 2:31:03

      D over DX, right? And they generally align, right? The exact Gelu and the, um, approximate Gelu generally align. And guess what? You can also do integration, integral of minus infinity...

1313. 2:31:18

      Oh, did I spell it wrong? Oh, infinity to infinity, right, of... I think this works. I'm not 100% sure, right? You take your exact Gelu, you minus the difference.

1314. 2:31:32

      Oh, I don't think, so this works. I, I, I ... Yeah, I don't think so. Oh, yes, it works. Yes, it works. So what you do is you can take the integral of minus infinity to infinity, so the entire line, your minus exact Gelu and your approximate Gelu, and you do DX, and there is a difference, right?

1315. 2:31:46

      But the difference is very small, right? It's like 10 to the minus 16. It's very, very small. And notice, it's like when you do, when we do fast Triton kernels, I generally use this feature.

1316. 2:31:57

      So do do ... You can do integration, integration and derivatives, and, you know, you can use Desmos. So I highly recommend Desmos. Um, and if you do this, that's where we found the problem.

1317. 2:32:10

      It's like, oh, okay, there is some sort of issue, and if you fix it, remember, the Gelu fix does do some effect. It does do some effect. But remember, we only showed there was only very small effects, so it's not that useful.

1318. 2:32:22

      Um, the RoPE fix was the most important, right? The RoPE fix actually caused issues, um, so you must fix that, and that's the most important fix that you must do.

1319. 2:32:31

      Um, and finally, there is, like, some other things that we do. Um, depending on the precision that you use, there is actually a difference between float 16 and bfloat16.

1320. 2:32:40

      And if you do this, we show that float 32, um... Remember we showed before that in the fixes that we did, the lines sometimes go back up Right? But actually, if you do float 32, it actually does work.

1321. 2:32:55

      If you do float 32 precision, the lines actually don't- do separate very well. But once you use float 16, the lines then match up again, right? And bfloat16, the lines match up again.

1322. 2:33:05

      Right? So this is just a phenomenon that you're using fast, smaller precisions, and that is why you have this problem. But if you do use full precision, you get good results.

1323. 2:33:15

      And the fine-tuning notebook for the Gemma [REDACTED:generic_id] also works. So Gemma is [REDACTED:generic_id] times faster and uses, like, I think 60% less memory as well. It's more now. Um, so if you run this, remember you have to connect to, um...

1324. 2:33:26

      You have to connect to your Google account, and you will get this to run. Any questions on the Gemma [REDACTED:generic_id]?

1325. 2:33:33

      Okay. Yes.

1326. 2:33:35

      Sorry I came in a little late. Can you just summarize the graph you just showed and,

1327. 2:33:41

      uh-

1328. 2:33:41

      Okay

1329. 2:33:42

      ... lines over number of layers?

1330. 2:33:44

      Yes. Um, where did I put the picture? Oh, oh, there. It's in the blog post. Um, yes, that's fine. Um, wait, where did I put it? Oh, it's the first picture, right?

1331. 2:33:55

      Yeah. This [REDACTED:generic_id], right?

1332. 2:33:57

      Yeah.

1333. 2:33:57

      So the X-axis is the layer number. So Gemma has 18 layers. So each of those, the X-axis just indicates which layer the, um, which layer it is. The Y-axis is log2 lay- log2 norm.

1334. 2:34:11

      Um, log- log L2 norm. So what you do is you take the original implementation, like DeepMind's implementation. You take Hugging Face, PyTorch, Gem... Like, you know, the other implementations.

1335. 2:34:21

      You check the output of both of them. So the output, you run the model through. You take output to layer [REDACTED:generic_id] and output to layer [REDACTED:generic_id], the o- the o- other implementations, and you just find the error.

1336. 2:34:31

      Um, and so this is just the error. Um, and this is log scale. Um, so when it's log scale, it looks better. When it's not log scale, it looks very bad.

1337. 2:34:39

      Um, so does that... Is that better?

1338. 2:34:42

      So this, you're, you're taking the output of each layer?

1339. 2:34:44

      Yes, output of each layer. Yes.

1340. 2:34:48

      Thank you.

1341. 2:34:49

      Um, that's called Gem- So that's for Gemma. For Phi-3, um, similar. What you do is you open up the Phi-3 implementation. You read through the Phi-3 implementation. And because, like, you guys, like, prob- most likely can go through Llama and, like, just look at it in general, remember, delete useless parts of the code, you will see there

1342. 2:35:08

      are differences in Phi-3. Um, and the differences are, um, they use other methodologies. Um, they use upcasting, they use stuff. But there was a weird thing that we found in the config file.

1343. 2:35:20

      Um, I will show you, um, Phi-3 config.

1344. 2:35:29

      Okay, I'll just use the instruct version. Um,

1345. 2:35:34

      if you go to... Always, when you go to, like, new models, always read the config file, right? Config.json. When you open it up, it tells you all the tricks you need to know about the model architecture.

1346. 2:35:45

      Um, and I highly... Right, it tells you what is the EOS token ID, 32,000. Right, when you look at this, hmm, is that a good idea? 32,000. 32,000. What is the EOS token ID?

1347. 2:35:55

      Right? 32,000. Okay, that's fine. The pad token, null. Hmm, is that a good idea? Like, you have to think about, like, why are these... Why are they there? How many layers does Phi-3 have?

1348. 2:36:07

      It's 40, right? So 40 layers. Um, how many positional encodings does it have? So how long... What is the context length? It is 131072. That's the context length. Um, remember, it's 100...

1349. 2:36:19

      So this model, the Phi-3 medium, is 128k, right? It's not 1- 128000, right? Just be careful. It's actually 128k. Right? It's 131072. Um, there are other issues with this model as well.

1350. 2:36:33

      Um, okay, this is... Okay, that's the... Okay, that's probably... Okay, probably don't use the instruct version. The instruct... Oh, sorry, the... We choose, choose the small version.

1351. 2:36:42

      Um, this is the smaller version. There is a thing we noticed. There's a sliding window. So Mistral has sliding window. Um, sliding window essentially attends to only 2048 tokens.

1352. 2:36:52

      Um, and this just makes training much faster. Um, and does anyone notice what the problem is for this? Why is it 2047? Anyone notice any issues?

1353. 2:37:02

      Yes.

1354. 2:37:03

      Well, it's not a power of [REDACTED:generic_id], but I-

1355. 2:37:05

      Correct. So is that weird?

1356. 2:37:07

      I mean, that's horrible, yeah.

1357. 2:37:09

      So what I did ask the Hugging Face people, and they said, "Yes, it is a bug." So they actually did fix it, but then I don't know why they reverted it back, so I'm a bit confused.

1358. 2:37:17

      Um, they never... They kind of forgot about this.

1359. 2:37:20

      It's a process.

1360. 2:37:21

      Yeah, so it's actu- it's supposed to be 2048. Um-

1361. 2:37:22

      Yeah, because that's, that, that only makes sense, 'cause you're training on 48, like, you know, [REDACTED:generic_id], [REDACTED:generic_id]... You're training on the correct context, right? Then this sliding window makes no sense.

1362. 2:37:32

      In fact, I've seen a lot of sliding window bugs recently.

1363. 2:37:35

      Yeah, so-

1364. 2:37:36

      For some reason.

1365. 2:37:37

      Yeah, I, I'm, I'm... Yeah, I'm not sure why. Um, but I'm pretty sure this should be 2048.

1366. 2:37:41

      Yeah, I'm very confident it's-

1367. 2:37:44

      I, I'm actually, I'm 100% sure it should be 2048. Yeah, it's not-

1368. 2:37:48

      It's very obvious

1369. 2:37:48

      ... yeah. And yeah, so these, these small issues are, they need to fix.

1370. 2:37:53

      Yeah.

1371. 2:37:53

      Um, they still have not fixed. Um, but when Unsloth, it's fixed, so we actually uploaded models which fixed them, right? So if you go to, uh, Unsloth Hugging Face repo, we actually have models which we fixed all of them.

1372. 2:38:05

      Oh, this is too big. Um, where is the Phi [REDACTED:generic_id]?

1373. 2:38:09

      Oh, I didn't put it up. Okay, I need to find the Phi [REDACTED:generic_id] now. Um, where's Phi? Oh, there. Phi-3 mini 4K instruct. Right, if you go to files, you go to config.json, we fixed it.

1374. 2:38:23

      Uh, and there's other things that we did to fix it. Um, for example, the pad token ID is [REDACTED:password]... Okay, that's actually wrong. Okay, hmm.

1375. 2:38:28

      Yeah.

1376. 2:38:28

      Okay, I need to fix my own... Okay, anyways, um, there is a bug which we discovered ourselves. [laughs] It should be [REDACTED:password]. This is actually wrong. Another thing is you must not make the pad token the same token ID as EOS.

1377. 2:38:39

      Never, never, never, never, never. Um, this must be a different token to the EOS token. Um, I do know Unsloth will automatically fix this during the loading. It's just the config itself is not right.

1378. 2:38:50

      Um, but that's okay. Unsloth itself is fine. Um, just the config is a bit wrong. Um, oh, okay, I found my own bugs, but okay. Yes.

1379. 2:38:59

      Oh, yeah, yeah

1380. 2:38:59

      I'm not gonna slow down the, the ... I'm not ... You keep going, 'cause there's a lot of slides

1381. 2:39:03

      Okay. Uh, yeah, yeah. Actually, there's ... Okay, there's not that much slides. Okay. Actually, there is. Oh, okay. I just noticed we have more. Okay. Um, so another [REDACTED:generic_id] is, like, Phi-3 used, um ...

1382. 2:39:13

      They merged the Q, K and K. Remember we did Q, K and V? They're unmerged, right? The weights are separate for the attention matrices. Phi-3 did a very interesting move, is that they fused them into [REDACTED:generic_id] matrix, and we found that to be very problematic for fine-tuning, um, because if you fuse them together, um, when you do

1383. 2:39:30

      LoRA adapters, you actually tr- you actually only learn new extra weights, and it's very less. Um, so please unfuse them. Um, and we do this ... So our version of the Phi-3 actually unfuses the weights.

1384. 2:39:43

      Um, you must unfuse. Actually, I, I have to, like, highly suggest you to unfuse the weights. You can only fuse them if you wanna do training faster. Um, this will make training, like, maybe 5% faster.

1385. 2:39:52

      It's actually not that much. It's, like, [REDACTED:generic_id], 2%. You actually increase memory usage a lot, so just be careful of that as well. Um, oh, yes. They actually did ...

1386. 2:40:00

      So this is the sliding window [REDACTED:generic_id]. They actually fixed it, um, and then they unfixed it. Uh, I think they just forgot about it. I'll probably, like, push them again to fix it.

1387. 2:40:07

      Um, and this is the fusing of the weights. Um, so we show that if you actually unfuse the weights, so Q, K and V must be separate. You must not combine them.

1388. 2:40:16

      Um, if you combine them, you actually have lower accuracy, so please do not do that. Um,

1389. 2:40:21

      for tokenization, remember this slide we just showed you about the, um, the smiley faces are the spaces, and each [REDACTED:generic_id]'s a different tokenizations. There are actually many issues, um, for tokenizations.

1390. 2:40:31

      Um, this is a totally different separ- separate topic from finding bugs and issues in language models. Um, this is a whole topic of its own, because tokenizers are very problematic, um, and they're very hard to find and fix.

1391. 2:40:43

      Did I double this slide? Okay, I doubled that. Um, and also we have new Ollama support which we have not announced yet, which you can try out. So lots of people have asked us for how do we actually fine-tune a language model and export it to Ollama, um, effectively.

1392. 2:41:01

      Um, does any ... Does ... People do know what's Ollama or no? Or does anyone know- not know what's Ollama? Okay. So Ollama is, like, a interface. When you, when you fine-tune a model, you have to run it, right?

1393. 2:41:12

      But you have to run the model somewhere, and Ollama just makes you run the model much easier. Um, so, like, you know ChatGPT? ChatGPT is, like, the running mechanism.

1394. 2:41:20

      Ollama's just like ChatGPT, but they don't have the model. You have to select a model. Um, that's kind of Ollama. Um, yes?

1395. 2:41:26

      How did you manage to, like, um ... Uh, so I've been working on converting, uh, creating model files using, uh, um, the, the, uh, automated s- pipeline, but we've been found many issues trying to automate model file creation.

1396. 2:41:43

      Um-

1397. 2:41:44

      Is this using Unsloth, though?

1398. 2:41:45

      No.

1399. 2:41:45

      Oh, is ... You using Axolotl or something, or other ones? Did you automate the model file yourself, or?

1400. 2:41:50

      Well, we ... Yeah. We ... 'Cause we need a ... We need our own model files, right?

1401. 2:41:54

      Oh, so we do this automatically now. So with Unsl- we actually ... We spent ... I spent, like, a few ... [REDACTED:generic_id] month on trying to automate the model file creation. [laughs]

1402. 2:42:01

      That's why we, we were struggling so hard as a company.

1403. 2:42:03

      Yes. Um, uh, we ... I have code for that somewhere. Um-

1404. 2:42:07

      And, uh, if you could ... Yeah, okay. If it's, uh, pu- open source, then-

1405. 2:42:10

      Oh. Yeah, but it-

1406. 2:42:11

      Thank you. Thank you

1407. 2:42:11

      ... it's already in the GitHub repo.

1408. 2:42:12

      Okay.

1409. 2:42:12

      So if you go to Unsloth, um, you go to chat templates, we have code for that, um, Ollama. It is still very ugly. Um, so these are the chat templates for ...

1410. 2:42:21

      Remember the BOS token someone mentioned, you have to add it? Um, yep, add the BOS token. This is the Ollama chat template, um, which we c- ... So Ollama has a pr- um, Ollama has a specific requirement, is you must have a chat template, because if you don't use a correct chat template, your model will output incorrect,

1411. 2:42:36

      like, substandard responses. Um, so this is the chat template for, like, some of them. I had to ... We had to write chat templates for all of the architectures, um, and we have an automatic [REDACTED:generic_id].

1412. 2:42:46

      So these are Vicuna and blah, blah, blah. Um, Alpaca-sket style. Um, Gemma, the Gemma style, we also have that. Um, we have many, many ... Even Ollama 3 chat template we have as well.

1413. 2:42:58

      Um, now, for the f- automatic [REDACTED:generic_id], so what we do is we can actually make an automatic chat template, a model file, for you automatically. Um, and this makes your fine-tune much more accurately.

1414. 2:43:09

      Um, wait, I'll show you the ... Where is the code for that? Um, where is the code? Okay, you can see the code is quite large for the ...

1415. 2:43:15

      Just the chat templates, right? This is just for tokenization. [laughs] So it's not even the ... Yes.

1416. 2:43:20

      This is Apache 2.0, right?

1417. 2:43:22

      Yes, it's Apache. Yeah. So it's, it's, uh, open source. Yeah. [laughs] Um, wait, where is it? Okay. So we have something called parse combined prompt which does some O of N squared.

1418. 2:43:33

      I didn't actually optimize this. It does O of N squared. I should have done O of N, but anyways, it's O of N squared, um, checking the prompt. Um, here's the prompt format.

1419. 2:43:42

      So we do ... It looks quite ugly, the code for automatic model file creation, but we actually made it so you can actually automa- it automatically creates a model file from your chat template.

1420. 2:43:52

      Um, you can see it's quite ugly. Um, but it works. Um, and,

1421. 2:43:58

      uh, yes. Oh, it's even more ugly. Yep. It's quite ugly code. Um, but unfortunately the model file is very hard to create automatically.

1422. 2:44:05

      Yeah.

1423. 2:44:05

      And so we have the notebook which allows you to do this. Um, so, so this notebook is in here, Ollama for Alpaca. So this [REDACTED:generic_id]'s for the Alpaca dataset, and so this is our installation, Llama 3.

1424. 2:44:19

      Um, where is it? So we, we're g- so we'll be using Alpaca GPT-4, um, GPT-4 dataset. So you use the Alpaca dataset, and you use GPT-4 to create the dataset.

1425. 2:44:30

      Um, and the trick is, though, um, we also have a CSV file now, so you can actually upload a CSV file and use Unsloth directly to fine-tune a language model.

1426. 2:44:40

      Um, and ... But the problem is a language model must have an instruction and output, right? Only [REDACTED:generic_id] columns. CSV files and Excel files can have many columns. So what do you do?

1427. 2:44:50

      You have to merge the columns into [REDACTED:generic_id]. Um, so remember each of those columns in your Excel file, convert them into text. And for example, the Titanic dataset- You merge them to say they have [REDACTED:generic_id] siblings and spouses and so, so on, right?

1428. 2:45:04

      You merge the row into [REDACTED:generic_id] row, um, and that's what you do. And with Unsloth you can do this now. It's... I still probably need to edit the, um, the, like, syntax calling, but this merging technique says, "Okay, your first column is called an instruction column, and the [REDACTED:generic_id] double brackets means it's optional."

1429. 2:45:23

      Um, so if, if the input column exists, then it will say the instruction followed by your input is. Um, and you can, like, make this very crazy. You can do as many columns as you like.

1430. 2:45:34

      Um, I don't know if the syntax is useful, but, like, I will probably be editing this. We're gonna make a YouTube video about this to talk about this. Um, this is actually very important for fine-tuning.

1431. 2:45:43

      Um, we noticed that every single provider requires you to use only [REDACTED:generic_id] column for instruction and [REDACTED:generic_id] output column. Now you can have infinite columns. Well, how many you like, but you must define the chat template.

1432. 2:45:54

      Um, and, and we also have a custom customizable chat template. Um, so before when you do fine-tuning of language models, you have to use the alpaca prompt, um, in our other notebooks, right?

1433. 2:46:05

      Below is an instruction to describe the task paired with an input blah, blah, blah. You put your instruction here, you put your input here, and you put your output here, right?

1434. 2:46:12

      But notice, what is the problem with this? Is there a problem with this?

1435. 2:46:16

      Only [REDACTED:generic_id] input.

1436. 2:46:17

      So you must only put [REDACTED:generic_id] instruction and [REDACTED:generic_id] output or response, right? The input is a problem, right? So how do you solve this? You solve this by merging the input into your instruction prompt, right?

1437. 2:46:29

      So this actually should be removed entirely, right? And your, your input should be something else. And what you do is we can actually... You're now-- We can do this now, right?

1438. 2:46:37

      So you must do... You must put the input, and you must put an output, right? You can only use [REDACTED:generic_id] columns now. But you can use, remember, even though you can only use [REDACTED:generic_id] columns, you can use this to convert your data set into [REDACTED:generic_id] columns.

1439. 2:46:49

      Um, yes?

1440. 2:46:52

      Do you lose any of the semantic meanings, though?

1441. 2:46:54

      Oh, no, I don't think so. No.

1442. 2:46:56

      You don't think so?

1443. 2:46:56

      No, I don't think... So it depends on how you, it depends on how you format the data set.

1444. 2:47:00

      Okay.

1445. 2:47:00

      Remember, it's a language model, so you can do... The more you tell the language model what to do, the better.

1446. 2:47:05

      Of course.

1447. 2:47:06

      Yeah. But the problem is, to do the model file creation, you must do [REDACTED:generic_id] iterations, repetitions of this, right? You must do instruction response, and then you do another [REDACTED:generic_id].

1448. 2:47:15

      You must, okay? You must do this for Unsloth. I found this to be very, very important for the model file creation. If you do not do this, you have dangling new lines, and you actually make your model output terrible.

1449. 2:47:22

      Um, so you must do [REDACTED:generic_id] repetitions of this, okay? It's a must, must, must. Um, and if you don't do that, we'll error out. Um, and so once you do this, we also have examples of...

1450. 2:47:31

      For example, this is Llama 3's chat template, right? We again do [REDACTED:generic_id] iterations. [REDACTED:generic_id]... You must do [REDACTED:generic_id] iterations. Most important. Um, and when you finish training the model, um, remember you can do runtime run all.

1451. 2:47:44

      Um, you can do inference now, right? Continue the Fibonacci sequence. Your input is [REDACTED:generic_id], [REDACTED:generic_id], [REDACTED:generic_id], whatever, and the next number of Fibonacci sequence is 13. I think that's correct.

1452. 2:47:54

      Yes, that's correct. Um, so your language model has learned how to do Fibonacci, and because it's a chat template, you can also do... You can shove in multiple messages into the model.

1453. 2:48:04

      Um, so this becomes a ChatGPT for you. This, this is a customized ChatGPT that you can use. Um, and finally, when you wanna save the model, um, you can save it to LoRA adapters.

1454. 2:48:14

      So this only is [REDACTED:generic_id] hundred MB in size. So once you fine-tune the model, you have [REDACTED:generic_id] hundred MB. But some people also want to, like, merge the model back, um, and that will take sixteen GB.

1455. 2:48:25

      Um, but you must merge this for, like, um, Ollama support and GGUF and stuff like that. And what we're showing for Ollama support is you first have to, like, you know, install Ollama.

1456. 2:48:35

      Um, you select what you want to save the model to GGUF, so this is eight bit. Um, we now support multiple quantization methods, right? You don't have to do eight bit.

1457. 2:48:44

      You can do, like, four bit, five bit, whatever you like, and this will be saved into [REDACTED:generic_id] go. Uh, much faster. Um, in fact, I think this will save you, like, twenty minutes of your time.

1458. 2:48:52

      Um, and we save this automatically. Um, okay, and this does all the saving, blah, blah, blah. It saves. And we also, you see, we automatically create an Ollama model file automatically using a chat, chat template.

1459. 2:49:04

      And I can verify this is actually correct, because I tried it. Um, and then when you wanna serve the model file, you can actually print out the model file which we created, and this is the model file.

1460. 2:49:12

      Um, whoops, I pressed run already. Um, anyways, um, and finally, to serve it, you can just do model file to serve it, um, and you can serve this. Um, and we do have a CSV version, so you can actually use the Titanic data set.

1461. 2:49:27

      Um, okay, it's loading. Um, so if you wanna use the Titanic data set, you can upload the Titanic data set, right? We... I upload the Titanic CSV. You can use the CSV, CSV file for this.

1462. 2:49:38

      Um, and again, you have the merger columns, and so on, right? This is a more complicated example. Um, in fact, I sh- pr- provide this entire example for you for the entire Titanic data set to merge all the columns into [REDACTED:generic_id].

1463. 2:49:49

      Um, and it's the same exact output. So that's the notebooks that we're sharing for... We do not release this yet, so this is for you guys to, like, experiment and see if there's any issues.

1464. 2:49:58

      Um, yeah, and just tell me. Um, we also have blog posts on our website, which you can see, um, our Unsloth GitHub repo. Um, and we have stickers available, um, and they're very, very cute for you to take.

1465. 2:50:11

      And thank... Yeah. And also, yeah, we have Q&A now. Yeah. Yes. Oh.

1466. 2:50:16

      Did you measure, did you measure the difference that it makes to write the CSV content in English sentences as opposed to just dump the... or put them in JSON format?

1467. 2:50:30

      The problem is, if you put them in JSON format, you still need to have instruction and output. So how would you do that?

1468. 2:50:36

      You need to have [REDACTED:generic_id] columns only for fine-tuning.

1469. 2:50:39

      Can you do, like, you know, your, your... In, in your template here, you have instruction, and then you add all-

1470. 2:50:48

      You, you add all of the other columns onto it.

1471. 2:50:49

      Can you just put, put the same instruction and then JSON of the, of the values of the columns?

1472. 2:50:56

      Yes, you could, you could do the JSON file. Yes, you can. Um, but we just show you that you can do multiple columns now. So, like, if you have, like, ten columns, you can now make the ten columns into [REDACTED:generic_id], um, by merging them together.

1473. 2:51:08

      Does that kind of-

1474. 2:51:09

      There's a big difference in representing that merged columns as an English sentence or, like, a dictionary?

1475. 2:51:16

      Oh, no, you can't use... You mean, like, you shove in the actual dictionary for fine-tuning? I mean, you could do that. I don't... I think you should do English language, because a language model predicts the next word.

1476. 2:51:26

      JSON is probably less useful. Always convert it into English.

1477. 2:51:31

      I have the same intuition. I was wondering if you measured it-

1478. 2:51:34

      Research paper. Yes. It should be another research paper. Yeah. Any other questions? Yes.

1479. 2:51:40

      So there's a lot of upvoted questions from me on the chat. Sorry, it's, uh-

1480. 2:51:45

      Oh, there's a-

1481. 2:51:46

      I was wondering if you could take a look at them. I don't want-

1482. 2:51:48

      Oh, yeah. Oh, yeah. I didn't actually check the Slido questions. Whoopsies. Um, it actually didn't load. So, ooh, there's lots of questions. Okay. I will... Okay. Oh, okay. Oh, okay.

1483. 2:51:57

      Oh, okay. I need to, I need to, um, answer each of them afterwards. I think I'm already out of time, though. So, yes. [audience applauding] Thanks a lot. [outro music]