Training an LLM from Scratch, Locally

[upbeat music] Thank you very much for joining for this workshop.

Uh, this is gonna be a hands-on workshop, so hopefully you can get your hands dirty doing some very interesting, uh, very interesting project. Um, to start with, a little bit about myself.

Uh, my name is Angelos. I lead the speech-to-text team, uh, at ElevenLabs. I'm a research engineer, so I spend the most of my time training new models, uh, working on inference, and also I'm on the product side, so I'm, I'm responsible for, unfortunately, talking to clients sometimes, uh, which I don't fully enjoy.

I'm the kind of person that really likes to go straight into research and do, uh, train models and make things that are state-of-the-art and very powerful. Uh, currently I'm working on, uh, training real-time models for transcription, uh, specifically for agents.

And if you have-- if you don't know already, we have the Scribe v2 model that my team trained. It's currently the best, uh, transcription model in the market in terms of like popular public benchmarks.

Uh, so if you have a need any transcription, uh, use cases, feel free to use it. It's, uh... I think it's quite good. Uh, now, as part of the workshop itself, uh, today we're going to be training an LLM from scratch.

So no pre-trained weights, no, uh, nothing that you can just grab online from like a transformers library. We're going to work purely on Torch and some, like, very basic libraries.

We could go on level below and, and not use Torch, but I don't want to, to torture you that much. Uh, so I think Torch is like a good level.

And this will be like a good indication of how like actual like, uh, research engineers like in, in big labs like design their models and

whatever is further than that is more like optimizations and making the scale be better and bigger and improving for specific use cases. So this, what you're gonna be doing today is like pretty much like eighty percent of, of the way there to, to create like a model from scratch.

Um, if you can go on this, uh, QR code, uh, you're going to find this, uh, GitHub repo. Uh, I'm going to switch to that now. We'll leave it for a little bit longer.

Uh, there's-- you have two options. Uh, one is to either train the model locally on your laptop. Like, if you have like a sixteen gigabytes of memory, you should be able to do it.

It's a small, very tiny model that you can train fast. Uh, if not, and I'm guessing because not many people have outlets, maybe a Google Colab would be a- another option.

Uh, Google Colab gives you free GPUs that you can use for, for training. It's a small model. So either, it will be your choice. Uh, while... I'm gonna leave this on for a little bit longer,

and then we can go in the actual repo itself.

Um, so to give you some idea of the inspiration of this, um, uh, o-o-of this project, uh, my first exposure to, to transformers and LLMs in general was through this video from, from Andrej Karpathy, who was one of the co-founders of Com-- of, uh, OpenAI, uh, which was called nanoGPT.

And for me that was like a very inspirational project. It's essentially like what inspires like this, this workshop as well. Uh, it's a bit lower level than what we're gonna be doing now, so it goes a bit deeper, like into how you're using NumPy for like a lot of ca-calculations.

But I think it's a good, uh, blueprint that we can follow for, for us to be able to create a model from scratch.

Um, let me move this to the screen.

But yeah, if you, if you don't, if you, if you, if you don't know this project, uh, I think it's, that's, it's a great introduction. If you do know this project and you've do- trained LLMs like that before, uh, this might be like sim-- a bit simple for you, but I think the, there are ways we can

expand this workshop to-towards, uh, doing something that's a little more of a competition, which I would see or like to see what you can come up with. For this specific, um, for this specific workshop, we're gonna work with a very small model that's based on the GP-2-- GPT-2 architecture.

Uh, it's a bit of an older architecture, but the, the, the base and the, the fundamental parts of it are basically haven't changed too much. Uh, and we're gonna go over, over those in, in a bit.

Uh, the, the four building blocks you need to, to train a model is first one is gonna be the tokenizer. Uh,

and depending on what your use case is, you'd wanna use a different tokenizer for that specific use case. If you want to train, for example, a very big model that can, that you want, you want to generate text for like multiple languages, you'll need a huge tokenizer, which means you're gonna need a huge amount of data as

well to train it. But for a s- for a smaller model, smaller tokenizer with, with, with, uh, smaller embeddings is what would work best and train the fastest when you're da- when you're data limited, which is what we are right now.

Uh, next would be a model architecture. Uh, to be honest, like most models like have at least at, at the, at the period that we're gonna be working in, were very similar.

Like in, in they were just decoder-only, co-causal decoder-only models that had a very similar, uh, way of, of using causal self-attention and the same like MLP layers and the same like layer norms a-and all kind of stuff, which we'll, we'll get, we'll get to it.

Um, so i-if you know how to do this, these like small models, it's very easy to go and do the same process for like bigger, like newer models. But of course, the newer models are going to be way more specialized for longer context and essentially, essentially being able to scale this train.

They're, they're architected in a way to be able to scale training to as many tokens as possible, which we won't need in this case. And lastly, it's a, it's a training loop, which this is generally the most important part when you're training a new model.

Like if you check the difference between like GPT-4, GPT-4o, and GPT 5, or even, even if you go before that,

what you'll see mostly is that usually the pre-training is very similar. It's the fine-tuning and post-training, and essentially what you use with the, with the same base model or like very similar base model, and the way you train it that actually makes the big difference in performances.

And now we see, for example, uh, Gemini III comes out and then has this, this, this many, this good benchmarks, and then three point one comes out that has like double the performance in some benchmarks, which is crazy.

Like, obviously, it's very sim-- the models are very similar, but actually during training, they train, they, they train the new model in smarter ways to improve performance like very substantially.

Now as for the... And, and lastly, of course, the inference part, like that's gonna be very easy for us, like because we're just-- it's gonna be a small model that can run everywhere.

Uh, so like that, that's gonna be very simple, uh, part of the, the building blocks.

Uh, then the prerequisites, any laptop would do, uh, that has at least six-- sixteen gigabytes of RAM. You can work with, with, with smaller laptops as well. It'll just be a bit slower.

Bigger laptops, you can crank up the batch sizes higher, so it will train faster.

Uh, Python th-- three point twelve, and I expect that, I expect that most of you have like some idea of how to write Python. Uh, like if you don't, I think it's still quite...

You can just copy-paste things until something works or like ask for some help. Uh, this training can-- uses, uh, Apple silicon, so the, the MPS architec-- the MPS architecture, CUDA or CPU, so you can bas-basically support everything.

Um, and as for getting started, uh, this, just to make things easier, like I'm using UV for this project. Uh, so if you have your laptop out, you can install UV on your machine if you haven't.

Uh, it's, it's quite straightforward, and there's no-- UV is quite simple. You can just run UV sync, and it creates a virtual environment for you and makes your life easier.

And we're gonna write the code in, uh, Scratchpad, or if you're using Google Colab, which depends if your internet is not very good, maybe that's a better idea. Uh, if you just go to,

um... If you just go to and create a new, um... Let me actually create it on a separate one, so you can, you can follow along.

Yeah. So if you open Google Colab, and you create a new, um, Colab project, you can run this command,

which just installs the, what we need, which is just Torch, NumPy, TQDM, and Tiktoken. Tiktoken is mostly for, uh, for testing things out.

And yeah, we're gonna move on to the first part of this, um,

uh, of this workshop, which is gonna be the tokenizer. As I mentioned, like this is generally the first thing you think about when you create a new, a new, any new transformer model, is what tokenizer you're using.

Uh, I come from the voice world, so this is where that-- that's the one of the most important things. Like we're thinking, "Okay, we need to train this new TTS model, and we're gonna spend maybe six months thinking about the tokenizer, and then we're gonna spend two months on the architecture."

So that's, that's like generally one of the most important things of deciding how to create a new transformer. For those who don't know what a tokenizer is, like LLMs don't see text.

They, they work with embeddings or like vectors. Uh, so we need like some kind of representation of those vectors for the model to be able to process.

Uh, what we're gonna be using here is a character-level alignment, uh, a character-level tokenization, sorry, just because it's, it has the lowest number of possible tokens. Um, in the, in our case, for on our, uh, dataset, it's gonna be only sixty-five embeddings essentially 'cause it's se-- sixty-five different characters that will appear in the, in our training data.

And the way it works, uh, like this, it's gonna be using the Shakespeare dataset. There's a few works of Shakespeare. It's part of the, um, it, it, it's part of the repo itself, but you can-- if you're using,

um, if you're using Colab, there's gonna be a link on how to download it on Colab a, a little bit later.

And essentially, we're gonna use this, this story library, which basically just converts, uh,

we-we'll be converting strings to integers, and that these integers will then be turned to embeddings through the embedding layer when we train the, the LLM. And it's a very, very simple, straightforward tokenizer.

It uses this enumerate, uh, function from, from, uh, from Python and just then selects that specific item that has been selected with, uh, uh, this dictionary.

So yeah, as I said before, like we use, we use character level because it's much easier to train. Uh, because we have sixty-five, sixty-five, uh, tokens only, it means that we-- the bigram combinations will be like sixty-five times sixty-five, so four thousand two hundred and twenty-five possible bigrams.

Bigrams are essentially like when you have one token, and then you predict the next token after it. And this concept of, of, of bigram is very, is a very important concept when, when you're training transformers 'cause you want your model to see as many possible bigrams as possible.

So if you have a, a model with, let's say, two hundred thousand tokens, you need two hundred thousand tokens squared at least data to be able to, to train like a, uh, from scratch in a, in a, in a, in a very good way.

Uh, so the-- or, or at least, like this is the magnitude that you're looking for. In our case, four thousand bigrams is like very doable. Like this, this dataset should include it.

All bigrams like multiple times very likely. Um, if we did try to train using a full tokenizer, this will never converge. We can just be training it for hours and hours, and then our model will never be able to, to get good results.

Now, the problem with, with character-level, uh, tokenizers is that they don't really scale very well. Um, because the models, they, they-- the way they work is that they need to understand correlation between different tokens.

Um, so you can very easily have a correlation saying, "The sky is blue." Like, these tokens combined together, like, make a lot of sense. But S-K-Y, and then I-S, and then B-L-U-E, that it's a bit harder for them, for the model to be able to make, like, good, uh, uh...

To attend to these tokens in, in, in a good way. Uh, so this will, this will work quite well for our example. But if you want, like, a very, very good model, uh, one, it will be expensive to train because, of course, like, you have to create, like, a ton of tokens when-- during inference and during training.

Uh, but also, it will just never converge to something good 'cause the model... The, the, the tokens combined don't make too much sense. Um, so that's our trade-off, but it's a trade-off that we're willing to take because, of course, we're running a, a, a small model.

But in the future, if you want to expand this to something better, if you want to train, like, a proper LLM, and you're happy to train for, like, a week or using bigger GPUs, uh, using a proper, uh, tokenizer, that's, uh, used, like, byte pair encoding is, like, the most common, uh, way of, like, doing tokenizers these

days, which essentially takes all... You can-- The way you train a BP tokenizer is that you take all your training data, and you'd find all common patterns and combine those common, uh, th-those common, like, character patterns into specific tokens that then you can reuse, and the model can understand relationships of.

So the, the way that this connects to the model itself is we have this embedding table, which is size, vocab size, and then it's what I said before. It takes, uh, a vector of integers, and then it returns...

Uh, sorry. A, a list of inters-integers and then it returns a list of vectors, which is going to be our embeddings.

Uh, and like, like I said before, like, that's... I-if we did use, uh, such a big, such a big tokenizer, that would also be, like, more than twice the size of our models.

'Cause if you just multiply this, our, uh, embedding size is three hundred and eighty-four for this model we're gonna be training. So that's already like twenty-five thousand parameters, and in that case, like, if we used G- uh, GPT-2's vocab, uh, which is fifty thousand, that would be nineteen million parameters, which is, like, more than three times the

model, which wouldn't make sense in our case.

Uh, so now moving on to the next part of this workshop, which is gonna be the transformer itself. And as I, as I mentioned before, like, the transformers, they have been kind of commoditized now.

Uh, they are-- The way the transformers work, like, there are different labs that find different optimizations. But the optimizations, like, in principle, is more about we have this, like, base, like, idea that works really well.

How can we make it train faster and, like, have bigger context and...

And people find ways that are more optimizations than necess-- than, than necessarily reinventing the wheel. Uh, for this, at least, there are some, of course, hybrid approaches that are more complicated.

Uh, so I, I'm not gonna go too deep on how transformers work. Uh,

and also to prove a point that you don't necessarily need to know in a very deep level how transformers work to, to be able to train something like this.

Uh, when I first, like, did this project myself, I didn't... I had no clue how transformers worked, and I still didn't that much at the end of the pro-- the worksh-- the previous project I worked on.

But it-- Once-- The more you work on it and the more you, like, you have motivation to continue pushing through, you can understand all the different concepts together and how they, they, they stack together and the reasons why they ended up the way they are.

Um, to go back to the big picture, uh, transformers are,

are u- are using, like, these four different building blocks. Uh, one is multi-head self-attention. Uh, attention is, like, what... The difference between-- that makes transformers different than other neural networks is that they can actually attend to previous tokens and understand the relationship be-between tokens that I was, I was mentioning before, and that's where a-attention comes in.

And of course, the bigger your attention is, the more the b-model understands tho-those relationships. And going back to what I said, that's what, like, big labs like Gemini, they're trying to do one million contexts, and they're finding ways.

Because if you just try to use a one million context for, for a model like this, it will just break very easily. Like, the, the math wouldn't work. So then that's when, like, the engineers from-- the researchers from Gemini found ways to, to make it work, and that's what...

That, that makes a difference. But again, fundamentally, it's, like, the same, the same architecture. Uh, next one is the, is the MLP, uh, or, like, the feedforward network, uh, which essentially takes these

different, uh, um, uh, relationships between those tokens and

while taking them... W-while the, the, the relationship themselves are, um, arranged all o- The relationship between the tokens themselves, it, it combines them together to, to be able to generate the logits that will then be, uh...

So it's essentially kind of a... It takes the context and then organizes it in ways that the model can then generate the logits and then generate the tokens. I, I hope that was a decent explanation.

And then you have the residual connections, which are basically there for the model to,

to not have to reinvent itself after every layer. Uh, re-residual basically means that every layer that pa- that passes through the, um... 'Cause as, as, as we'll talk about later, transformer is m- is m- is, is built on multiple different layers that we pass through the activations one after the other.

And, uh, residuals are there so that each activation doesn't completely, uh, restart things from scratch. It just changes them slightly. So it just takes the previous input and makes a small-- another small difference.

The next layer does the same, the next layer does the same, and this will continue on. And this way the model doesn't--

th-each layer doesn't make a huge change on the inputs themselves, uh, and the model can be more stable during training. And lastly with the ra-layer normalization has a very similar, uh, has, has a very similar role in, uh, being able to...

The, the, the layer normalization is able-- is, is allowing you to scale down those activations in ways that,

uh, that allows that-- those activations to not be, to not be exploding into very big values. So if one layer multiplies your activation by ten x, let's say, the layer norm is gonna push it ba-back to like normal values, so it doesn't go ten x, ten x, ten x, and then you can have like t-- millions of,

uh, values that start at like zero point five and then end at ten million. That's what the layer norm is for. Uh, but again, these are just the building blocks.

You don't necessarily need to know like why they're there and what the, what the purpose of them is. Like, you, you learn this as you start working on these models more and understand, um, how those dif-- why, why these decisions were made.

'Cause all of them, uh, uh, as, like, as I'm talking to you, of course, were done by-- We have a certain idea. All this didn't work. So let's add this to make it work.

Sorry, go ahead.

Are you showing some slides that we aren't seeing?

Oh, oops. Yeah, I was showing.

No worries.

Yeah, apologies for that. Yes. So, um, I was, I was going through the tokenization. Uh, that was the previous, uh, what, what, uh, my, my previous point. Uh, and that's, uh, if you, if you can go back and, uh...

You'll have to go back through and forth through the slides when you're working on your-- the model yourself 'cause you can-- you have to like copy-paste the par-- the, the parts or figure them out yourself.

But it's-- it was explaining why we're going with character-level tokenization and, um, and what other options we have. Then for transformer, I was explaining the different in the big picture, the different blocks of, of architecture.

Um, and then now going to how does this look like in code? Uh, 'cause all the things I described to you actually are very co-- there, there's very little code you need to, to act-- to implement them.

Uh, first we start with the basis of... Sorry, go ahead.

Um, when it comes to code as in Python, and because at least in English, you know, you have words, vocabulary is kind of fixed. But in Python, you have your own variables and so on.

So how does tokenization work in that setting? What could be at work? Sorry.

So in a programming language, uh, of course, you have programming syntax, uh, keywords, but then you have also normal variables, function names, and so on. So when it comes to tokenization, how does that work?

So yeah. A-as I mentioned to you, to-- before, uh,

we're gonna be using like a character-level tokenizer because it's gonna be easier for this project. But most big labs, they don't use character-level tokenization. They, they use what I was mentioning earlier, the BPE tokenizer or like, uh, byte push, um, uh...

Essentially they, they-- what they do is that they-- you u-u-- you look at your training data. Let's say you have like this many trillions of tokens, and your training data is going to be including like code, as you mentioned yourself.

Uh, and the way it works, it will just look at common patterns. Uh, so if you have like a lot of training data that is, uh, that is code itself, it will, it will see, and then it will realize, okay, for loops seem to be like a good candidate for, for that to be a token.

So the for is gonna be for sure a token. And then you might have some like enumerate. Enumerate you f-- do you see quite commonly in the, um, i-in the training data, so that will be another token.

So it will look at all these different tokens and then create this tokenizer based on the common, uh, relationship b-between them. Uh, of course, like maybe some languages that are like maybe Pascal that might not be very common in the language, so this-- the keywords of Pascal might not be like very common.

Although I do think that there probably is like good representation in the tokenizers too. But some o-- other of those like crazy like white space only like languages that these, these ones probably are not going to be work very well. [laughs]

My, my question was more on the... So one thing is the keywords of the programming language, but then you have your own variable names, which are not common among different programs.

Ah. I see what you're saying.

So making them as tokens, I'm not sure how does that help.

So a-again, like, there, there's no, there's no specific way. There's no like human in the loop in this process. It depends on your training data that you use to train this tokenizer.

Uh, if your code, like there's like foo and bar, like that's like common variables, these probably are gonna be tokens. Like if your, your to-- if your variable names are like very strange, like random characters, then yeah, probably that's not gonna be in the tokenizer.

And when this happens, uh, the tokenizer will fall back to character level, uh, or like, uh, b-bi- uh, bytecode level tokenization. So if your, if your token is like just random like gibbles, it would just be each character or like some-- maybe some combinations might be together, uh, as different tokens, but most of them are gonna be

separate tokens, and that's gonna be a bit of a pain to-- for doing inference and, uh, yeah, maybe not, not, not the best if you want, uh, efficient inference.

Thank you.

Um, but yeah, uh, going back to, to the transformer side of the things, um,

as I said, the, the models are generally quite, uh, qui-quite similar. Um, and the code for actually implementing these transformers is also Quite simple and, like, easy to write.

It's, like, maybe, maybe a hundred lines. Actually, usually more than that, uh, less than that. Um, and the first thing we have to re- to, to, uh, accept is what the parameters of this transformer should look like.

Uh, as I mentioned before, vocab size is the size of your tokenizer. In our case, it will be sixty-five. Block size is essentially the sequence length or, like, the, the, uh, the context window.

Um, in our case, that will be two hundred fifty-six, which is very, very small for these models. But because we're training a model locally, like, that's kind, kind of what we have to do.

Uh, bigger labs would use, like, one million context size for stuff like that. Uh, but generally, like, sixteen thousand is, like, a common middle ground. Uh, then we have the layers.

How many... As I mentioned before, like, uh, transformers have multiple layers, and then you run activations through each one of those layers. We're gonna go with a modest number of six.

And then the attention heads, like, the, the way that, uh, a-attention works is that you might... You've ha- gonna have different heads for attention that are gonna be attending for different things.

Like, one of the attention heads might be looking at punctuation. Maybe another attention head might be looking at the grammar. Uh, so all these different attention heads are attending to, like, a specific, uh, feature of the text, or if you're using audio, a specific feature of the audio, et cetera.

Uh, and lastly, that's the embedding dimension. Uh, uh, that's how big the actual vectors of those tokens that you, uh, create are. Uh, in our case, we're gonna start with, uh, three eight, three eighty-four.

That's, like, the standard for GPT-2. But a, a bigger value, of course, will have more information per token. A smaller value would, would have less, but that's, like, a pretty standard value we can start with.

Um, as for the code itself, uh, feel free to copy-paste part of this, uh, if you wanna spend more time understanding it. I don't wanna go too deep into, like, all the little details of how things work.

Uh, but essentially, like, you, you generally have, like, this overarching module. In our case, we're gonna call that GPT, and that overarching, uh, top-level module is gonna include all the other modules that I described earlier.

Uh, in this case, th- this will take the config.

Uh, in this case, this, this will take the config that we have, we, we had above, and then we'll create this, uh, using-- We're using torch, uh, module dict here just because it's, like, the easiest way to implement this.

But this is all just math. Like, you can... Everything you see here would just be, like, calculations, like two matrix multiplications that, uh, Torch allows us to abstract and make things easier.

And pretty much all the things here are li- are you see here are, like, neural networks, like, smaller or bigger that are combined together. Uh-

Can I ask a quick question with regards to these parameters?

Yes.

If you could scroll up. So you have, uh, um, the length of the sequence, right?

Yes.

Let's say we have a use case where we have-- we don't have a large sequence, right? So we have however many. Two fifty-six is fine. But this, uh, is very much related to the number of parameters you have in your model.

Um, not necessarily. It, it... Sorry, maybe you can repeat the question with the microphone.

So the, the, the size of the input sequence is related to the number of parameters in your model.

It, it... Not, not necessarily. You can have, like, large sequences with, like, small models or, like, small sequences with, with, uh, with, uh, bigger models. The, the, the main thing that this shows is essentially w- there are two ways you can train a model.

You can train a model, like, that looks at... That's a full attention model, let's say, that looks at the full attention, so it looks at the whole past, uh, essentially, uh, which is what we're building here today.

Or you can have a windowed model. That's where most of the, like, bigger models that are out right now that only look up to, like, this much in the past.

And this parameter is the size of that window that... Okay.

Yes. It, it's what we're gonna be using to train. So the model w- will not have seen anything greater than two hundred fifty-si- fifty-six tokens in a sequence. If you try to go above that, it will kind of, it will spaz out.

It will have issues.

But so let's say we have, like, computational capacity to increase the number of parameters to make the, help the model reason better, right? Which number would you crank up here to say, like, would you increase the number of layers, attention heads or, um...?

I, I would increase all of the, all, all of what you see here, except for maybe the, the embedding, uh, size and maybe dimension. I don't remember exactly what other models do, uh, the exact numbers.

I think that seems reasonable to me. Uh, but everything else, like, it works well for, like, a small token, uh, a small, a small model. Uh, two fifty-six block size is tiny.

Like, that's... If you try to run this on nanoGPT, like, it would just forget things that you wrote like ten sentences before. Uh, but

the, the bigger you make this, the harder it is to do the training. And that's what I was talking about scaling laws, is that you can't just say h- go here and say, "Okay, actually, I don't want two fifty-six.

I want two million context length." You can't train this like that, at least with this current architecture. So that's what, like, uh, uh, like, researchers after, like, GPT 3.5 came out, they were like, "Okay, people are complaining that we only have sixteen K context size.

We want one million. How do we do it?" You can't just change this number. You have to change the architecture to allow training to be able... Otherwise, you're just gonna go, like, out of memory, like, instantly.

Uh, so that's, like, one of the things that researchers now try to, to, to figure out, how we're going to be able to increase these numbers here while keeping training stable.

Um, or how, how do you increase these numbers here while keeping training stable and still able to, to have enough c-compute to be able to do this? I hope this answers your question.

Awesome. Thank you. Um, so yeah, the, the first thing we'd, we'd have to do is, uh, as I mentioned before, like create the, the top-level representation of our model, as you can see here as the GPT class.

Uh, I'm not gonna go again too, too, too much into the details, but like you want, uh, one would be the embed- how the, your model like understands embeddings and also how does it understand embedding position.

Uh, I don't wanna go into details, but essentially you need-- the tokens need to be able to understand both. Uh, then you'd have essentially here all the layers that I was mentioning earlier, and each of them is configured as, as a block, which we're gonna go at the definition of block, uh, late, uh, a little bit later.

But essentially a block is like a layer of the transformer that includes its own attention and, uh, its own like layer norms, and it continues the-- it feeds into each other the, the one you do the forward pass.

And lastly, you have the LM head, and essentially the LM head takes all the outputs of the above and connects them together into what we call like logits, which is the distribution of what the, the next, uh, token should be that should be generated.

Uh, 'cause I, as I mentioned earlier, like may-maybe I did, maybe I didn't, but, uh, the way transformers work is that they predict next token. So you take the previous con-- the previous, uh, context, you predict the next token, and you have to sample this token by on, on, on this distribution.

And the LM head creates this distribution that we, we sample from.

Then the, the, the next important bit that, uh, like is again like quite, quite straightforward, like and is, is quite similar in most of these models, is a forward pass, uh, which essentially allows the model to,

um, to go-- to, to take the input, which is gonna be the tokens that we mentioned, and push them through all these different components that, that, that I, that I mentioned earlier.

Uh, and just does some, um, uh, does, does some, some preprocessing, turns tokens into embeddings and positional embeddings, uh, add, uh, uh, adds them together, then goes through all the blocks of the transformer, all the different layers of the transformer, and just runs the, the forward pass of that transformer.

And then does the linear norms and then passes through the LM head to get the distribution of those logits that we mentioned earlier. Um, and this is if you're doing training.

This would also do cross-entropy loss, um, in this case. And then at the end, you're gonna get logits and loss as the outputs of your, of, uh, of your transformer model.

And I do have some diagrams here that you can see. Uh, like they're quite, they're quite basic, but essentially they show the flow of take token IDs, you pass them to make them into embeddings, token embeddings and positional embeddings.

You add them together, you pass them to the transformer, then through the layer norm that just makes the outputs, uh, in a space that the LM head can comprehend easier.

And then you return this distribution of like what the next token should be that has, that has this, uh, this size. And as we mentioned before, like sixty-five is the size of like your voc-- our vocabular-- or our, our token size.

And now self-attention is like a little bit more complicated. I don't wanna again, again to go too deep of how, how that works. But essentially att-attention is there to, to understand the relationships between the tokens.

Uh, uh, essentially, what, what is the important... Like if I say the sky is blue, blue and sky have a, a very big correlation. Uh, so that's what attention does.

Based on the, uh, how you've trained your weights, you'll understand what tokens should be attending to each other and put higher emphasis on those specific relationships. Now becau-- again, going back to what I was sa-saying before about the, the what-- that's why the tokenizer matters a lot, 'cause sky and blue are very easy to-- uh, very easy

for the model to make this relationship. While in our case, you have to like combine different groups of tokens together, the different characters together. That's quite a bit harder.

But essentially, that's, that's what attention does. It's, it's the what, what this token should be attending to in the past and what has most importance for it.

And it has also a forward pass, as all of these different blocks do. And in your implementation, feel free to just copy-paste these, these blocks. Uh, maybe you can spend some more time on understanding the diagrams and, and how things work.

Uh, but as you can see, even something as complicated as attention has a very small... It's just a few lines of code to, to implement.

And yeah, lastly, the, the MLP block, uh, is again, as I mentioned earlier, uh, al- also like again, I already talked about the multi-head atten-- uh, why attention has multiple heads, 'cause each head attends to different parts of what makes language what it is.

And then the MLP block takes the outputs of attention and, and makes sense of all those different relationships into something that the model can then understand a bit better, um, which again is just essentially just a, a neural network that just combines things into something that's a bit more understandable for the LM head to then be able

to, to generate distribution for logits.

Sorry.

Yes.

Can I interrupt you? Because I just have a question.

Yeah, no, you can interrupt.

Do we have the same MLP in all, uh, individual transformer blocks?

Uh-

Like, is it the same architecture and the, the, the same weights between the different transformer blocks?

No, each, each block has its own weight. Each block, uh, it, it-- the, the way each layer, essentially you'd have each layer that we'll call blocks. Um, they usually have like a different section for, like a different prefix for the weights of what...

Like MLP would be a different, uh, like maybe usually they're called FFN blocks, uh, instead of MLP, but it, it will still be the same. So each block has its own weights.

And, uh, the ML- each, each layer-

The different

Yes, each block has its own MLP. So it's, it's-- you ha-- each block has its own, um, uh, as you can see here, its own attention, its own, uh, linear le- li- linear norms and its own MLP.

That's what basically makes a layer of a, of a transformer. And, and that's what I-- what my point was gonna be later. Everything comes together into this block that we can see here, which is basically a layer of a transformer that has some normalization, running the attention to get the relationship between the models, and then having the

MLP that combines the relationship of the, uh, the... Sorry, the relationship between tokens, and then the MLP that takes those relationships and turns them into like a representation that's easy for the model to, uh, to create the logits.

And, uh, this, uh... Oops. This is what I was showing. Sorry. I, I have two different screens. I shouldn't be doing that.

But yes, this is, this is the transformer block. This is the, the building block essentially of a, of a transformer. Uh,

it has layer norm, the attention, and another layer norm. Like, not all of the models have this, this kind of configuration, but this specific one has this. And then the MLP that takes everything and creates it into a representation that makes sense for us to be able to make the generations.

And, uh, you can see here like a bit more of a, like a simple diagram of how this, this works.

Yes?

Did the original, uh, nanoGPT have the residual connection as well? Do you remember?

Uh, so the residual should be there. Um.

No, no. I mean on the, on Andre K- Andre Karpathy's nanoGPT. Did, did it have the-

Uh, I don't, I don't remember. Uh-

Okay

... it's been a while since I-

Okay

... like I ki- I, I had the idea but from there, but I built everything from scratch. So I don't, I don't really remember if it did or not.

Uh, but the residual, as you can-- the, what-- the idea of residual, uh, would be, as you can see here, instead of doing X equals the attention, you do X equals X plus attention.

So you, you get the difference. The, the, the values of the activations don't change that much. Uh, that's the idea of like doing residuals. Um,

uh, and yeah. So we have this, like if you wanted to implement this yourself, you can copy and paste all these different classes into one file that you can call model.py.

Um, and this is essentially all the maths of how the transformers works. Um, and as I mentioned before, then we have to decide how big the transformer is. Like the parameter count that you can see here, uh, is just the ten million parameter based on what I was showing above.

Uh, you can just find it yourself by just summing all the, all the different parameters of all the different, uh, parts of the different blocks that we had above.

Um, the token embeddings is sixty-five times three hundred and eighty-four. We said this is the vector for its embedding, so it's twenty-five K. The positional embeddings is two hundred and fifty-six.

Two hundred and fifty-six, remember, is the sequence length, the max sequence length of the model. So that's another ninety-eight K parameters. And then where the, the biggest part of the logic is, is in the actual transformer blocks themselves where most of the parameters live.

So you have, um, f- you have four X. Uh, so the, the four there comes of the way attention works, where you have key value, uh, key, q- query key and value pair.

So it's, it's part of the potential has like four different parameters for, for, um,

for three hundred and eighty-four, which is the amount of, of tokens, times the relationships between those tokens. So two hundred and eighty-four per vector times three hundred and eighty-four.

So that's five hundred and ninety that we have for the, for the attention per layer. And then for the MLP, you have again the similar logic of, um, don't remember what this one thousand five hundred and thirty-six is.

Uh, but, uh, yeah, in the end it turns out into being one point two million. So total amount of parameter size we're gonna be training is gonna be one point eight million parameters, which should be good for-- which should be easy to train for like most devices.

Uh, so yeah, you can go back to this. It has a lot of details that, that makes, um, uh, that, that will, that will make sense for you to go back and understand a bit deeper and do your own research on, on this.

But this is like the general like very, very high level idea of how these transformers work.

And, uh, we're gonna go now to the, the training loop,

and that's where most of the meat of this project is going to be. Um, where we have this, this transformer, uh, how we're going to train it and do what we want to do.

The objective of this training we're gonna be working today is something that one has to be very easily recognizable. Uh, like you know when the model starts working and, and groks, which means that it, it understood what it's trying to do.

It's gonna be very easy to understand that the model actually works now. Uh, and secondly, it should be something that's like quite easy for the model to learn. Uh, if you teach the model how to, to write, uh, Python or like a competitive programming for you, then yeah, that's a, that's, that's a, that's a very hard task,

so we're not gonna be able to do it here. But in, in our case, like the, the objective is going to be to create a Shakespearean-like LLM that can create, uh,

verses from, from Shakespeare. Um, and as we mentioned before, the, these models like are, are learning as next token prediction. So

the, the way that cross entropy works is that you take your current, the current tokens that you want to train. In this case, let's say that it's T zero, T one to Tn.

And then you want it to predict T one, T two to T plus one, T, Tn plus one. Uh, so the way this cross entropy works is that you take your sequence and you just offset it by one as like what the model needs to learn what to calculate.

And of course, the last one you don't have it, so you have to, like, cut it off. Uh, and then the model learns how to predict the next token based on, based on this logic.

Uh, now for the actual training code itself.

Um, first we need to load the data, and we have this function that, that loads the data. The data is already in the, uh, repo itself. It's in this da-data dir-directory.

It's a, it's a collection of different, um,

different, uh, like lines and verses from, from Shakespeare. It's, it's about one million tokens, so like a million characters.

And we first load the data. The data loading is, is, is, is very simple. Too simple. So that's one of the things that I think you can optimize. Um, essentially just takes all the tokens, uh, splits it into a, a validation and a, and a training set, and then just essentially shuffles it and takes

part, uh, two hundred and fifty-six batches, uh, of sequences of, uh, of, of, of tokens that are the-- from, from the text itself and, uh,

and just essentially uses it for training. Batch size is gonna be sixty-four, so it takes two hundred and fifty-six, uh, token sequences, sixty-four of them, stacks them together and passes the, the model to, to, to teach it how to, how to train.

So a very simple data loader. Usually these data loaders are-- can be quite complex, um, because of course, like if you have-- especially when you have higher context lengths, like the way you load data is like a very big part of how the model like list needs to learn.

But in our case, we have a very, very simple implementation.

Um, next up is the way that w-- th-this is how the code works for, um,

we need to g-to use a device. Uh, this code works for MPS, it works for CUDA, and it works for CPU. Depends on what your l-your laptop supports or like if you run things on Google Colab.

Um, like if you run Colab, it will detect CUDA, and that'll be quite fast. MPS is also quite fast. CPU will be the slowest, but that's like a-- that still would work decently well.

Uh, next up is the learning, the learning rate, and that's like a big part of how the, the models are able to learn. Uh, the way the models work is that you first start generally with as high of a learning rate as, as you can, as you can afford without making the model

b-become unstable. So it, it basically makes it go crazy. Uh, and the way it works, you start with a very high learning rate, which is essentially the, the amount of the model is able to learn per step, how much the mo-- the weights need to move into the direction that you want.

If you have very high learning rate, you would offshoot your target, so it can go off the rails very fast, your model. So you want to use a very appropriate learning rate, um,

for your model to be able to train well. And usually you have this, this concept of a warm-up, where you start with a very small learning rate so that all your optimizations, uh, the, the model weights are able to st-to, to stick to places that are, um, that again, that they don't go into like...

They, they can, they can start in places that are appropriate for the training to begin. So you start a very low learning rate, increase it slightly unti-until you reach the peak, and then at the peak you continue-- you start reducing the learning rate.

That's what we call weight decay. Um, and, uh, until you reach the point that you're, you're satisfied with it. Some people re-- For some people this is zero. Like I prefer to not go to zero 'cause then it's hard to restart the training.

Um, but that's the idea. You want the learning rate to be less when the model is like close to being perfect. So small-- You, you want it to start with very big changes to find like good local minima, lo-lo-global minima, and then it-- for it to calibrate as training goes, goes further.

Um, so we'll start with like a small warm-up of a hundred steps, and then we're gonna use a cosine decay until the max steps that we're gonna... In this case it's gonna be five thousand.

So you're gonna start from like very low, peak at one hundred steps, and then start going down, down to, um, to five thousand steps. And that's what the AdamW normalizer does.

Uh, it essentially allows this, um, th-this concept of like controlling the learning rate, um, using this cosine decay. And that's the most common normalizer people use. At least used to use.

Now there's better normalizers, but this is the most simple to, to start with.

And here's how the full training loop would look like. Again, like not, not, not that much code. You just,

uh, initialize your config. Uh, like in this case it would be like six, uh, six layers, six attention heads, three hundred and eighty-four, um, embedding size, and then two hundred and fifty-six, that's the sequence length.

Uh, you initialize your model and then you create your, um, you, you start your optimizer, you start your steps. TQDM helps with like tracking like the losses and, and all that kind of stuff.

You want your loss to start high and then keep going down until you reach levels that are acceptable.

Um, and then another important part is evaluations to make sure that your model actually works well. That's what the val loss, uh, is here. So

it's very easy for models to like overfit, especially like this, this, the small models, 'cause you don't have that much data. So the loss might keep going down, which means that the t- what the model predicts and what the training data is are very similar.

So your loss can keep going down a lot. Uh, but actually at one point, maybe your model actually overfits in this case. And when it overfits, even though the loss goes down, actually the performance of the model is worse.

That's why we have this val loss Which is a, is a part of the data the model has never seen, and we run a forward pass to get the loss of that specific part of the dataset.

And if this is very low, it means the model can-- is, is performing well because you ha-- the model has never seen this, this data, so it can't memorize things it had never seen.

So that's what, that's what we have here with the VAL loss.

And, uh, I'm not gonna explain the concept of, of, of, of, uh, of, uh, backwards losses, but essentially that's the way the model tra-- the model weights move towards the direction of being optimized.

Uh, so for, for each step, we have a batch size of, in our case, two hundred and fifty-six, um, two hundred and fifty-six tokens, batch size sixty-four. So we ha-- we push this matrix through our model, and then it learns, and then the optimizer does an extra step, and now the learning rates are adjusted depending on what

the steps you're in right now. And lastly, just to have an extra way of, uh, e-every, every one thousand steps, you save your checkpoint to be able to restart your training if you need it from that point.

And we're also running, uh, inference on the current checkpoint to see what the model actually predicts at this point. Uh, so what we're, we're gonna start seeing is

when we start from s- from be- the beginning, because this is a model that we train from scratch, the loss is going to be essentially random. And in that case, that will be natural log of sixty-five, so it will start at around four point seventeen.

Uh, that basically means the model like knows nothing. It has no clue of what this data is. Uh, and slowly, we're gonna start seeing the loss going down to three point three.

That's when the model is gonna understand character frequencies. Uh, it will still not be able to do words yet, but it might understand things like T-H as part of "the," which is a common word.

Like T-H is gonna be part of the things that it starts generating. Then at around two point five, it's gonna un-- it was gonna get a little bit better about this T-H, and then it will understand the word "in" and stuff like that.

Then at about one point five to two losses, it will start actually creating words. And then at around one, one point zero to one point two, that's when the model is going to start being decent at, at, at this task.

It will actually be able to understand names from the text. It will start creating things that start making sense. But then when the loss starts going below one point zero for this specific dataset, that's where we're gonna start seeing overfitting.

The model will still be like producing like reasonable things, but it will no longer start, uh, be getting better at it.

Uh, this is like an example. At two hundred steps when I was testing this, uh, it was just producing like complete nonsense.

Uh, and the VAL loss was around three point five. Uh, then at about eight hundred steps, it started producing like decent things. Still not, not, not great things, but it was, it was starting to get there.

And at a thousand steps, it got better. And there was one point that the VAL loss actually started increasing instead of decreasing, and that's where we know the model overfit.

So at around two hund-- two thousand four hundred steps is where the-- that was the optimal performance of this model. And then if we kept going, the, the performance was actually maybe not decreasing, but the model started becoming less creative.

So that's one of the things to, to keep in mind. Now, VAL loss is like not the best metric. Like if you're actually being serious about training LLMs, usually you might have like some benchmarks that are running as part of your training, and you can see like if the benchmarks are like getting worse or not.

Uh, but for us, that's like a very easy and cheap way of like understanding how the model is doing.

Um, so yeah. That-- what, um, next steps now, like, uh, would be to actually train this model yourself. Um,

considering the internet is not very good, I would suggest probably using Google Colab for this.

Um, you can, you can copy-paste stuff from, from, from, from here. You can glue things together. Uh, copy-pasting will probably like w- allow you to g- get ninety percent there.

Uh, there is a lot of room for, for, uh, improvement for what I written here. Like on purpose, I made it like super simple, and there are things that you can improve yourselves.

Uh, but the idea is to, uh, to get something working. I hope everybody will be able to get something working if, if you're interested into, into, into working on this and have something that, uh, starting from nothing and getting like a model that can actually produce like a result that seems reasonable.

Um, after we have trained the model, the next part is text generation, which is like the inference side of things. Uh, we're just gonna be using, uh, there's multiple ways to, to do, to do inference.

One way is like greedy decoding. It's what I was mentioning earlier. You have all the logits, which is just a distribution of tokens, and you just take the most likely token, and that's what greedy decoding means.

Uh, so let's say that you have like your-- the token T and the token H are, are both in the distribution. One has like eighty percent probability, the other one has like fifteen percent probability.

You always be taking the, the top, the top one. Uh,

that's greedy decoding. Greedy decode doesn't work very well for LLMs. It can work well for other models, but for LLMs, like this essentially makes them very boring and not very creative in what they generate.

So you pretty much never want to use greedy decoding for LLMs. You would want to use it for, for other, um, models. Like transcription, for example, greedy decoding is the best because there's usually like only one way you can, you can s-- transcribe something.

You don't want it to be creative in transcription. That doesn't, that's not a good idea. Um, so that, that's what greedy decoding is. In our case, we're not gonna use that.

We're gonna use temperature. So essentially what temperature is, is that you're not always choosing the highest probability token. Sometimes you might choose the second highest probab-probability token or like the third highest.

And, uh, even though it doesn't make sense, why would you choose like a worse token for this situation? It has proven that this actually makes the model perform better.

Uh- It re-- like, sometimes it might, like, go into, like, this weird loop, so there are, like, techniques you can make sure that it doesn't go, like, crazy by generating some nonsense.

Uh, the worst thing is if, if it predicts, like, an end of transcript token, an end of a text token, and then just, like, stops the generation, which maybe you have seen sometimes when using ChatGPT where suddenly it just, like, stops for no reason.

Uh, you-- sometimes it's because of that. But there are ways you can, you can, you can, um, prevent this. Um,

generally, like, a zero point seven, uh, temperature is, like, the best, the best middle ground to make the, the, uh, inference work well. And then you have top-k sampling, uh, which essentially, like, it prevents the model for...

I-if you have, like, let's say, five tokens that are very likely, and then the sixth token is, like, completely unlikely, top-k sampling pre-sampling prevents the model from predicting this suc- sixth very unlikely token even though the temperature might-- you might get unlucky and the temperature might hit it.

Um, so that's what top-k sampling does. And this is, like, our inference function. It's, like, it's very straightforward. It just does-- It just runs, uh, takes all the tokens as, as the input, passes it through the model, takes the mo- the logits out of the model, and runs, uh, soft...

This is the-- What I was describing before is called softmax. It takes using a temperature the probabilities, and then, uh, it, it decides what the, the next token should be based on, uh, on those probabilities.

Um, and one thing you can do, you can use seeds. And essentially seeds is that it, like, right now everything will be random if you just keep retrying it.

But if you, if you keep retrying it, but if you use a s- a set seed, then your inference will always be returning the same value, uh, which will be, uh, relevant in

later. Uh, then putting it all together. If we put all together, we should have three different files. One is, like, the model.py that includes our model architecture. One is train.py which includes the data- data set loading and the training loop.

And lastly, the generate.py that includes our inference. And in total, this should be, like, maybe a few hundred lines of code. Uh, ve-very straightforward, and that's with-- even with this much code, like, if we have a lot-- big hardware, we could train a good LLM using this architecture if we have enough data and enough resources.

That's, that's all you need, essentially. And that's what basically, like, GPT-3 and GPT-2, like when OpenAI released it. I remember when OpenAI was about to release, uh, GPT-2, and they were saying, "We're not gonna release it because it's too dangerous for, for humanity."

And that was, like, back then. It-- The-- That was the code that they were working on and, like, a lot of data and, of course, a bigger model. Uh, now this seems to be, like, kind of, kind of funny, but for them, like, that was, like, a very wow moment that we did this, and then the model,

like, actually performs really well on tasks. Um, but in the end, it was just, it was just what you see here.

Um, so yeah, putting it all together, uh, you can, uh, if you use Google Colab, you can use this, this, uh, snippet of code to download the, the data set.

And you can use this pip install command to install the different dependencies. Uh, and then it should look, uh, like something like this,

where you install your dependencies, you copy your code.

Uh, and you might need to do some, um, connection, some connections together. But essentially,

like, you can run a, a train... That you can use a train command to run, to, to, and use the data set that you want to use, and that this will start the training.

And then you can see how the performance of the model improves step by step. Like, for me, this took about, like, fifteen minutes to train on, on Google Colab until I started getting good results.

Uh, with some improvements, you can make it to go faster, and maybe you can make it go slower but actually get better results. Um, but essentially, like, it's, it's very simple to get, to get working.

One thing to remember is that you have to change your runtime, change the runtime type to T4 GPU, um, because this, this, this is free, and it will, it will run quite fast.

Um, yeah, feel free to train. Like, you can try, like, maybe starting with a very tiny model, the zero point five million parameter, uh, with a-- only has two layers and only two attention heads and a smaller embedding size for, for each token.

And then you can try bigger and bigger models until you get some usable results. Um, and as I mentioned before, you can try different context lengths. I started with two hundred and fifty-six.

You can try bigger, five hundred and twelve. Um, and then you can, if you want to monitor and see how your losses went down, like, in a nice way, you can use this, like, uh, uh, this PyPlot lib, uh, to see, like, your, the graphs.

Uh, the things that you want to look for is if your train loss is not decreasing, it means your model is not learning. Um, that problem is you have a bug in your code.

Uh, if your, uh... It also could be, like, a, a low training loss. If your train loss is, is decreasing but your var loss is not, is actually increasing, that means you overfit.

Uh, and if you have very weird spikes in loss, like, the loss will be, like, very smooth in general. If you have very weird spikes in your loss, it means, like, again, there's some, some kind of bug either in your data or in your training.

And, uh, when the model starts plateauing and not getting any better, it means you kind of, you have pretty much exhausted the usefulness of your current data sets. So either you're gonna use a bigger model or you're gonna need-- you essentially need more data.

Uh, now one, one, like, interesting part about what, uh, I would like us to do is, uh, I think it would be cool if we had, like, some sort of competition of who can train the best model here if, uh, anybody's able to, to get things working.

Uh, hopefully the internet is good enough. Uh- And also for further reading, you can see like a lot of the resources for this, um, for this workshop.

Uh, but essentially what the challenge is, is

we can v- we, we can vote all together to find which, which model actually produces the best verse of Shakespeare, or it could be if you use a different data set, a best poem or in that kind of category.

Um, the rules are you have to train the model yourself, like f- here and today. Like it, it can't be like some... You can't ask ChatGPT to give you like a good verse.

You have to like use your own model. And to prove this, you'd use like a seed with a specific prompt and see what outputs it gives you. You're free to like regenerate things as many times until you, you get the best results.

Um, and, uh, you... I'll have like a QR code that you can submit your results, and I can go around and like help people out that, uh, if you need any help getting, uh, training running.

And, uh, of course, uh, this training is like super bare bones. There are many ways you can optimize this and make this better. So I guess for people that are a bit more experienced, they can, they can implement this, this, uh, improvements and maybe get a better outputs of your, of their model.

Uh, the winner is gonna get some, uh, free swag from, from ElevenLabs, maybe like a hoodie or like some, some free credits. Uh, I'll see what I can actually give.

So yeah, this is the submission. Uh, it needs to be creative, and then, um, and it needs to be essentially like a good verse. And we can maybe use like a kind of, um, a, a kind of bracket.

Oh, what happened? Oh, cool. We can use a bracket, and then we can vote which verse like sounded the best. They could be funny, they could be like, like well-made, like up to you.

And the reproducibility should look something like this, uh, where you run Python generate on your best checkpoint, the prompt that you decide, that you can... It's up to you.

Temperature, and then a, a seed that, that proves that you can actually produce your results. Uh, you can try different model sizes. Uh, like I guess like eighty-five million parameters would be a bit too big for this, but if you have the resources, you can try it.

Uh, you can try better tokenizers. Like this one, of course, is like character-based, but perhaps you can train your own BPE tokenizer, uh, based on that data set. And there's also other tweaks that you can do, like bigger context.

Uh, like there are sm- some training optimizations, like using a dropout value. Uh, you can stop whenever you want, when you feel like the model is good enough, change the learning rates, um, and essentially get like as-- make the model as good as possible.

Uh, yeah, that, that is the idea. Uh, I'm gonna go around and help you out. Sorry, go ahead.

In, in, uh, reasoning models, is it finding very different for training or is it similar but just some small, some more smarter?

So, uh, uh, reasoning models, like to, to repeat, uh, wh- uh, uh, you're asking like, uh, if reasoning models are quite a bit different in training. The base-- The building blocks are, are very similar.

Like you can train the same exact model. You can post-train it, which is how like usually reasoning is, is, uh, is being taught to this model. You have a good base instruct model, and then you post-train it to be a reasoning model.

This is very data set driven, so you need very, very high quality data, and you're gonna use a loss that's like good enough to be able to, uh, to learn this data in a, in a very good sense.

Uh, the complication of reasoning models is finding this good chain of thought data. Uh, that's why like OpenAI has like all these labelers that are like PhD students, that they li- write down the reasonings of like how to, how they solve problems.

Uh, because this data needs to be very high quality. 'Cause it's, it teaches the model how to think. So you can't just go on like Reddit and just get random posts.

You're not gonna learn how to think this way, for sure. Uh, you need like some very high value, uh, good quality data to teach this reasoning process. But in the end, reasoning is, is essentially just adding to the context of the model, like this, to, to the attention essentially.

Uh, this like logic that then the model can, when it generates a response, it can go back and attend to those reasoning tokens and get a better response out.

So it could be like describing the model a bit better. Then it goes back, sees those tokens that it described and say, "Oh, actually, yeah, I already figured this out.

I'm gonna write it down now." Uh, the microphone is not working.

Is it? Yeah. Perfect. Uh, so reasoning and non-reasoning models share the same base often.

Yes.

And then one is post-trained in a different way and adds this, uh, you know, in a way different, uh, post-training.

Yes. So a l- a lot of the labs, like let's say the Qwen model that was released, like Qwen three, usually they release like a base version, an instruct version, and the usually the base version doesn't have in s- a chain of thought reasoning.

Uh, it's usually like the same model that they first pre-trained to be like quite good, maybe do some fine-tuning as well. And then the next step is doing post-training to teach it this kind of like, this, this, uh, th-this knowledge.

Uh, so that's why like, uh, you see a lot of, a lot of, uh, big improvements that happen very fast in the industry. Like Gemini three to three point one, it's essentially giving it like better reasoning data and like, like better fine-tuning post-training data for like specific problems.

It's a little bit like benchmarking. Like usually this data is like very similar to what the benchmarks, the current good benchmarks are. But yeah, essentially just taking a base model, using this new data to improve it to the next level.

Thank you. And second question is, compared to what you're showing us here, like this very bare bone model, are there like, have there been fundamental innovation in the main training of the model, like powering nowadays models?

Or is it mostly the same, but just with Smarter tricks, small tweaks, better data, and so on. So I, like-- are everyone still using the same attention layer and so on, or are there like some fundamental shifts that have been happening on the latest?

And maybe you know, maybe you don't, I don't know.

Mm-hmm. No, yeah, no, uh, so you're, you're correct. That, that-- so they-- a lot of this base is the same. They might do some changes in terms of how attention attends to those different, uh, uh, tokens because sometimes reasoning can be quite large, so you need big to- uh, sequence lengths to be able to get good results.

Uh, so there's a lot of like tricks that the new labs, the, the labs do to make this, the, to, to make the attention more efficient. But overall, you can take like a very-- because you can take this model, GPT-2, and make it into a reasoning model.

Like, if you had the, the data and like a big enough model to actually learn from that reasoning as well 'cause like these tiny models, reasoning won't help them too much.

But if you had a big enough model that can get stuff out of the reasoning, there are people that have taken like older models, like let's say Llama one B, uh, that are small and weren't trained for reasoning, and that made them into reasoning models using the exact same architecture.

Yes?

How much effort do you put in, you know, getting your golden dataset, and what is that process that you follow?

You mean for post-training?

Yeah.

Yeah. So like, uh, as I mentioned before, like usually what most labs do is that they go to companies like Scale AI, and that's probably the biggest one. And Scale AI has an army of like people that you can, you can say, "I, I want physicists.

Give me data from physicists." Then Scale AI is gonna find contract physicists, pay them like as much money as they need, and then these physicists might like write things down.

They might be contacted to do different things. Uh, but essentially like a, a lot of these companies, they-- like Scale AI provides data for like Anthropic. Now they got bought by Meta, so probably not that much anymore.

Uh, but essentially they, they, this, these datasets are provided by these like, uh, labeling companies. Uh, Scale AI is one of them. But, uh, a lot of the, the big labs have their own labeling teams too, that they hire contractors to, to, to generate these datasets for them.

Uh, but as I, as I said before, like you want, you-- if this dataset has like even some small issues, it can literally make or break your model. So these datasets are like kind of the most expensive, like they will cost tons and tons of money, but they actually are the ones that make the models like as

good as they are.

Just, just a quick follow-up. So even on that,

I mean, you still have to evaluate, right? The answer wouldn't be exactly the same. So are people are really reviewing this or it's still you're relying on LLM to evaluate?

Uh, you, you, you rely on people for this kind of stuff. Usually, the way it works, uh, a big part of my job is actually like doing this, uh, organization.

Uh, usually, the way it works is you have-- you might have like one person that generates the la-- the, this like very high-quality label data. And then you have maybe some labeler that has graduated into like a QA position, where their job is to essentially make sure that all the like other labelers, the more entry-level, their outputs

are correct. And it's, it's, it's quite the tough job because like if your QA is not good, you get fired. So it's like, it's-- that, that's the way they keep the level quite high.

Yes? So LLMs are non-deterministic, but, um, how does that seed parameter work? So- So the, the way the LLMs are non-deterministic is because they, they essentially use n- random number generators for, of like the, of, of, of your machine, uh, or like of the, the, the, the GPU that you're currently using or the ma- system.

The seed essentially like makes all those different calculations to be always, return always the same value. So things are no longer random. Like you're-- it is not only for, uh, for, for this.

These seeds can be used for any random generation. Yeah. Uh, so it could be like for password hashes. In other systems, but I wasn't- Yeah, it, it works the same exact way.

Okay. Yes?

But this is like you mentioned greedy decoding, right? So if we use that, then they are deterministic, right?

Yes. Yeah, if you use greedy decoding, uh, there, there might be still some things that like, like it's good to use a seed even if you use greedy decoding 'cause there might be some other stuff that maybe you don't, you don't, you don't control.

Like maybe you're using some-- like sometimes greedy decoding might not be actual greedy decoding. It might be zero point zero one like, uh, temperature, like for vLLM, for example, that they do some like tricks like that to be able to get, uh, good outputs.

Uh, so in, in general, it's good to use seed even if you're using greedy decoding.

Yes?

Um, this is obviously all in text. ElevenLabs is mostly audio, right?

Yes.

How different is this versus like doing this with audio?

It's surprisingly very similar. [clears throat]

Really?

It's more complicated for sure, but parts of the, the stack of like most audio models, they also can do text. 'Cause they, you need-- the, the models need to understand language itself, and the best way to teach something language is text.

Uh, of course, you can train an, an audio-only model like, uh... And again, it's like what I was talking about the tokenizer, like how do you tokenize audio? Like what, what is the concept of like a sound?

Like how, how big should your tokenizer be for audio? What-- Should it be only human speech or should it also be like, uh, music? Like do you take the notes of different music instruments as like different tokens?

These are the kind of problems that you have to solve, like if you're creating a, uh, an audio model that you wouldn't necessarily need as a text, it's a bit easier.

Uh, but the fundamentals are still like the same. Like if you want to generate audio, you, you generate like an audio token, and then that audio token would be using a tokenizer, and then you, you...

It's not exactly the same 'cause the audio token then you have to process it in some different ways, uh, to make it into like actual audio. But, uh, a lot of the same, uh, general ideas still apply.

You know, like this is the text. The text can be down to very clear, like a, a one is a one, but in audio, like a particular pitch or frequency or sound byte, like there's the ambiguity there, so it's already lost straight away, right?

Yeah, yeah. So the, the w- the way ta-- it, it works is that you don't use like this cross-entropy loss. You use different types of losses. You can use cross-entropy, but usually it's, it's losses that are more, um, specialized for what you're trying to do.

Like for example, there's a ca-- a loss called L2 loss, which essentially takes like two mel spectrograms and starts to see the difference between those two mel spectrograms, which is essentially like, uh, sound waves in-- encoded in a certain way.

And that's like a very common way for TTS models to be trained. You train on these specific types of loss. Uh, and the same thing I was mentioning earlier, cross-entropy doesn't really work as well for, for things like post-training.

You might use different types of losses there. Or if you're distilling a model from a, a big model to a smaller model, you don't use cross-entropy loss. You might use a KL divergence loss where you find the, the token distributions of like the bigger model, and you try to match the logits of the smaller model.

So there's different types of losses for different use cases. Not all of them work the same way.

Yes?

So in a sense, you have like an audio vocabulary, right? And so you train on that. But, uh, what do they do with multi-modal models? Do they just like direct regular text vocabulary with audio vocabulary, with pixel vocabulary together and train that?

So the way mu-mu-multimodal models work, uh, usually you don't use tokens in the same sense. Um,

like the-- okay, this is where like it, it becomes a bit more complicated because like these models are not really built for this, like the GPT model. Like the b-- the newer models, they have also like an embedding input, uh, which essentially as, as, as I was mentioning earlier, like you have each token then corresponds to like

a vector. But these vectors, they don't have to correspond to a specific token. You can take these vectors from other places too. And what a lot of these labs do is that instead of having a tokenizer for like video, for example, what they do is they have another transformer that they call a video encoder, and they put

the video first through that video encoder. And, uh, this video encoder will be taking, let's say you have a, a thirty-second video. It will take one frame per second of this video, and it will take those frames and then put them through this like new transformer, this encoder transformer, that works quite a bit differently than this one.

And what you do is that you take the final layer of this transformer, the hidden values, which are also vectors. You take those, those vectors out of this encoder, and you're gonna input them in the embedding layer of the, of the transformer model.

So what the model is going to see is gonna... Usually it's, it's like prefixed. So you take the, you take the video, push it through the encoder, get some vectors, and then put those vectors in the embedding input of your transformer that does text.

And the way it would look like if you look at the sequence, it would probably be like a prompt, and then it would be probably like a video token representation.

And then actually-- But the embedding of the video token is gonna be overridden by the output of the encoder.

So that's how like these, these multimodal models work.

So the frame produces a vector which is the same length as like the vector for the horse.

Yes, it's exactly the same vector. Uh, the same for audio. You have an audio encoder and do the same but, but, but for audio. And, uh, but for-- in, in terms of how-- what the model cares about, it just cares about like these embeddings, right?

It don't care if it's text or if it's audio or if it's, or if it's video. It cares about these vectors, and that's how you, you represent them into the, the same dimension as the model, the transformer expects.

So, uh, there's, there's no relation between like a video of a horse jumping and the word horse. Is there no similarity between those two vectors?

Um, maybe there is. I, I'm not-- It depends on how you train it. Like it's-- these are the kind of things that are a bit about black boxes. Maybe there is, maybe there's not.

Actually, that's a good idea to like... Uh, that would be a good research paper to, to see, like, uh, for video encoders. Do they actually match in the same dimension as the text encoders?

I, I don't know. I imagine there probably is some, uh, connection. Sorry, you had a question as well?

Uh, yeah. I was wondering about like, do you do both, uh, normal speech, but you also do some music generation? Is that like a very different problem, or will the architecture be like similar?

And also like when you have these harmonics and stuff, are you still able to do just like a basic, like a autoregressive transformer, or do you have to like do diffusion because like things depend more on each other?

Uh, and you might wanna like generate everything at the same time even.

You, you, you can do both. There's music models that are autoregressive. There's music models that are diffusers. Uh, like it depends on how you train it. Um, like I think some of the, some of the Google models are like, like transformer-based.

Some of, uh, open source models are like diffuser-based. Like both can work, can work very well. It's just that it's, it's a little bit f- as, as I said, like it's a little bit more difficult to,

uh, to, to put into perspective this, the, the concept of like odd music. If you tokenize and predict the next token, it's kind of hard because it's very abstract.

Uh, so usually diffusers work a bit better in this like image modalities for generation or like music or like even audio for some models. Like they have some kind of diffuser, diffuser, um, diffuser part of the process.

Um, both can work. Diffusers are generally a bit easier to get it working.

I hope this answered your question.

Yeah, that makes sense.

Yep.

How do you, uh, tokenize audio? Because I imagine sort of with text, you get the repetitions, like repetition happens quite a lot. It's easier to combine those different tokens, create bigger tokens.

But with audio, you have distinct voices. Uh, I don't know, like how long, how long tokens would be.

It, it's very hard.

Okay.

And it's not really something that you do by, by just sitting down and thinking, "Okay, I'm gonna tokenize this word to that." It's, it's not something that you like sit down and like decide.

You use some kind of processing. You train like an audio tokenizer through... You have a d- I- it's a very similar like case I guess in, uh, how you train like a text tokenizer.

You'd use your training data and you'd find like common patterns in the audio and then tokenize those patterns. Of course, like when we say audio, we don't, don't always just mean like the actual like sample rate and like the, the a- the audio waves.

Usually convert them to something that's a bit more easy to, to tokenize, to, to do this processing. The most common one is like, uh, is mel-spectrograms, is first you convert your audio to mel-spectrograms and then you use this, this distribu- this, uh, um, essentially arrays of like, of numbers to train your, uh, your tokenizers.

And it will be very dependent on, on, on your training set. Like if you wanted to tokenize music and use a music data set, your audio tokens for music are gonna be very different than if you had a voice data set that's going to be like focusing on like human voice.

Now the hard part is what if you want to do both voice and music? That's where like... That's, that's very hard. [laughs]

Any more questions? Okay, awesome. Um, yeah, if you, if you want, you can start working on, uh, training the model if you have a laptop out, uh, or if you have already started.

Uh, you can follow this, uh... You, you can, you, you can follow the, the workshop, uh, try to get something working and, uh, if we do have enough submissions, we can do the, the competition and,

and, uh, see who- whoever wins who- is gonna get some, some nice pri- What's the cutoff for, for getting these things in? Sorry? What's the cutoff? Uh- Voting begins shortly after?

So, uh, uh, because we don't have that much time left, let's just say 5:45.

So if you have any, any questions or you need any help, uh, please call me and I'll, I'll come over. [outro music]