AI Engineer World's Fair 2025
How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS
Read the talk
How GPT-2 Works, in Vanilla JavaScript
Follow a prompt through tokenization, embeddings, attention and vocabulary scoring, using Ishan Anand’s browser implementation to connect transformer mechanics to familiar JavaScript.
From a talk by Ishan Anand
Before you start: Basic JavaScript and familiarity with matrix multiplication are helpful; no prior machine-learning background is required.
Can a web developer understand the machinery?
What do you need to understand a language model well enough to build on it? The apparent prerequisite list—calculus, linear algebra, a machine-learning course—can make the internals seem inaccessible. Ishan Anand’s workshop starts with a different challenge: take technology that looks like magic and expose its machinery using tools a web developer already knows.
His earlier project, Spreadsheets Are All You Need, implemented GPT-2 Small in pure Excel functions. One student, a CFO named Joe, came to the material through spreadsheet fluency rather than machine learning. The JavaScript workshop adapts that approach, compressing an eight-hour class into a planned two-hour session: ordinary programming and some familiarity with matrix multiplication are enough to begin. Python, React, Vue and advanced calculus are unnecessary for this walkthrough.
The working environment is GPT-2 in your browser, opened in desktop Chrome. The route through the model emphasizes its boundaries: tokenization and embeddings turn text into numerical input; the language head turns the final numerical representation into a token. Between them, attention and a multilayer perceptron transform the representations. Understanding those operations also supplies the foundation for explaining training and the later move from text completion to assistant behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Load the weights, then open DevTools
The model’s learned parameters arrive as CSV files: large tables of numbers, rather than an opaque executable. To load the workshop implementation:
- Follow the page’s link to the GPT-2 Small CSV archive on GitHub.
- Download and unzip the archive.
- Select all the extracted parameter files and drag them into the loader.
- Wait for the status indicating that all model parameters are loaded.
The loaded parameters occupy approximately 1.5 GB in IndexedDB browser storage. Chrome needs sufficient available disk capacity; this figure describes stored weights, not runtime RAM consumption. Once the files are loaded, the model runs locally, and Anand describes it as usable without an internet connection.
The interface resembles a notebook. One kind of cell contains executable JavaScript; another evaluates a formula and displays the result as a table. Matrices are ordinary two-dimensional JavaScript arrays. A formula can call a function such as getFinalTokens and reference promptToTokens, the DOM ID of an earlier table. The interface’s bracket notation resolves that table and passes its contents into the function. It is a small layer of table-reference syntax around JavaScript.
That makes the model available to the same debugging techniques used on a web page. Anand adds console.log(matches), reruns the tokenization cell and inspects its intermediate matches in Chrome DevTools. Then he inserts debugger and reruns the cell again: execution pauses inside the model, where variables and control flow can be inspected step by step. Removing the statement lets the remaining cells run normally. The useful property is inspectability: an unfamiliar model operation becomes a function whose inputs and outputs you can examine.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One prediction at a time
Give the model Mike is quick. He moves, and its immediate job is to predict the next token—here, quickly. It does not produce a paragraph in one operation. To continue generating, append the selected token to the input and run another prediction. Anand illustrates a subsequent continuation with and; repeating this feedback loop produces longer text or code. This is autoregressive generation. The browser workshop stops at one predicted token because that is the core operation it is exposing.
A sentence-completion problem becomes a numerical problem through a sequence of representations. First split the text into tokens. Then look up numerical vectors for those tokens, perform the model’s arithmetic, score possible output tokens and select one. Anand initially draws a simplified dictionary in which a word maps to a single number and an output is matched to nearby dictionary entries. That picture motivates the conversion, but the actual representations are vectors, and the output stage computes vocabulary scores. Randomness enters when a decoding procedure samples from those scores; greedy decoding instead selects the highest-scoring token.
GPT-2 dates to 2019, roughly three years before ChatGPT and four before GPT-4. Its release is often remembered through the phrase “too dangerous to release”; OpenAI’s announcement initially withheld the large model while releasing a smaller version. Its educational value is the continuity of its components with later transformers. Anand cites EleutherAI’s comparison with Llama 2 and uses a model family tree to make that point: learning this older, manageable implementation provides a foundation for recognizing the parts in newer systems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why tokens are neither words nor characters
A token is a vocabulary unit with an integer ID. It might be a whole word, part of a word, punctuation or a unit that includes a leading space. The running sentence happens to be easy to read as tokens, but shared spelling does not guarantee shared boundaries. When the network stalls, Anand switches to his local implementation and compares reindeer with reinjury. The latter includes a leading-space rein unit followed by jury; reindeer receives a different, three-token decomposition despite sharing the same initial letters. Tokenization is doing more than splitting strings at predictable word boundaries.
Why not give every dictionary word its own ID? Unknown words and misspellings immediately become a problem. An UNK token preserves the fact that something occurred but loses its identity. Unexpected languages and conventions matter too: Anand points to GPT-2’s rudimentary translation behavior and its ability to use TL;DR as a summarization cue. Retaining unfamiliar text makes such patterns available to training. A whole-word vocabulary also becomes large: his rough comparison is 170,000 English words against about 50,000 GPT-2 tokens, before adding other languages. More vocabulary entries require more parameters in the tables that represent and score tokens.
Character tokenization makes the opposite tradeoff. The example prompt occupies six rows in the model’s intermediate matrices, one per token. Turning every character into a token would make those matrices taller throughout processing. Individual characters also carry less immediate semantic information, leaving more structure for the model to learn. Anand uses the familiar jumbled-letter chain email as an analogy for recognizing larger units; its Cambridge attribution is part of the email, not a research result established here.
| Unit | Main advantage | Main cost |
|---|---|---|
| Whole word | Compact sequences of familiar words | Large vocabulary; unknown-word handling |
| Character | Fine-grained representation | Longer sequences; more structure to learn |
| Subword | Reuses common fragments | Boundaries do not always match human intuition |
Subwords balance vocabulary size against sequence length.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a vocabulary by merging frequent pairs
Byte-pair encoding, or BPE, separates two jobs: learning a vocabulary from a corpus and applying its learned merge rules to new input. It began as a compression algorithm in the 1990s. The learning objective is easy to see in Anand’s small corpus of low, lower, newest and widest: repeated words retain their frequencies so that common text receives a more efficient representation. Dots indicate token boundaries, and an underscore marks a trailing space. This teaching convention differs from GPT-2’s leading-space units.
Begin with individual lowercase characters, then count adjacent pairs. In this corpus, e • s occurs nine times: six times in newest and three times in widest. The frequency table identifies it as a most-frequent pair, so it becomes the next merged vocabulary unit.
The procedure then repeats:
- Add
esto the vocabulary and replace each adjacente • soccurrence with it. - Count adjacent pairs again, now treating
esas one unit. - Merge
es • t, which also occurs nine times, intoest. - Continue selecting frequent pairs and rewriting the corpus.
For example, the affected rows progress from n • e • w • e • s • t • _ to n • e • w • es • t • _, then to n • e • w • est • _. After ten passes in the demonstration, frequent words such as low and newest have become whole tokens. The vocabulary has grown while the number of tokens needed to represent the corpus has shrunk. Units such as low and est resemble meaningful language fragments, but the algorithm reached them by frequency counting, without understanding their meanings.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Apply the learned merges to a prompt
At inference time, the tokenizer uses the vocabulary learned during training. The browser implementation first applies OpenAI’s regular expression to split the prompt into preliminary chunks. In Mike is quick. He moves, punctuation separates out, while spaces remain attached to chunks such as He and moves. These chunks are not yet the final tokens. The implementation then loads vocab.bpe, which supplies the learned ordering of merges. Its display adds rank and score columns; early entries include space-plus-t and space-plus-a. A separate map connects resulting tokens to IDs.
The table for quick makes the operation visible. Start with its characters, inspect adjacent pairs and select the available merge with the best learned rank. In the displayed sequence, i and c combine first; later merges form qu and ick, eventually producing the full token. Inference reuses the learned ordering rather than building a new frequency ranking from the prompt. The unusually expansive table layout mirrors the Excel version so readers can follow the same intermediate states in either implementation.
This representation can complicate character-level questions. Counting the r characters in strawberry requires recovering spelling information from tokens rather than reading a sequence of individual letters. In Riley Goodside’s example, capitalization, leading spaces and quotation marks produce six different token patterns for variants a person readily recognizes as the same word. Tokenization can worsen these errors without being their sole cause.
The preference for subwords is empirical, not a requirement of all models: research systems also explore word and character representations. Nor must a token represent text. A Vision Transformer can use image patches as its units, while Anand cites Waymo’s use of spatial trajectories. The general question is what units make the input useful for the computation that follows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An ID finds a token; a vector represents it
A token ID does not itself describe meaning. GPT-2 Small maps every token—including a period—to a vector of 768 numbers. In the browser, each token becomes a row with the same width. Anand notes an implementation-specific limitation when experimenting with other prompts: the workshop importer can mishandle foreign-language characters because of a character mismatch during import.
Think of a housing table. A street address identifies a house, while square footage, bedrooms, bathrooms and price describe its properties. The token ID acts like the address; the embedding acts like the descriptive columns. A useful embedding places related tokens in related parts of a high-dimensional space. In Anand’s two-dimensional “word island,” happy and glad sit together, dog and cat occupy another region, and sad belongs near the emotions even though it differs from happiness.
A deliberately simple coordinate system shows why vectors are useful. Give the first coordinate the label authority and the second gender, with these teaching values:
| Token | Authority | Gender |
|---|---|---|
| man | 1 | 1 |
| woman | 1 | 2 |
| king | 2 | 1 |
| queen | 2 | 2 |
Then ordinary coordinate arithmetic produces a relationship:
The analogy becomes a mathematical operation: king relates to man as queen relates to woman.
word2vec made such relationships famous, including country–capital, profession and food associations. Its results were imperfect, but examples such as France–Paris, Japan–Tokyo and Italy–Rome illustrate a useful idea: a recurring direction through vector space can represent a relationship, suggesting Canada–Ottawa through a similar offset. Actual learned embeddings have hundreds of dimensions without the toy table’s straightforward labels. Even so, unlabeled columns can support comparison—just as similar numerical house records remain recognizable after removing their column headings.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn representations from the company words keep
Where do the embedding values come from? Training begins with random parameters, including random embeddings. Take an observed passage, Mike is quick. He moves quickly., withhold the continuation and ask the model to predict it. A randomly initialized model might produce something like haircut. Comparing its prediction with the observed quickly supplies a training signal. Backpropagation computes how parameters affect the error, and an optimizer uses those gradients to adjust them. Repeating this across many passages lets the model learn patterns of grammar, names and facts without individually hand-labeling those concepts.
The statistical intuition is that ice often appears with cold, while steam appears with hot. Even without knowing the words beforehand, an observer could infer relationships from their contexts. This is the distributional hypothesis, captured by the phrase “You shall know a word by the company it keeps.” Words with related uses tend to occur in related contexts.
In the longer class, Anand builds primitive embeddings from Wikipedia using a simplified, GloVe-inspired method. Count which words occur within a window—three words to either side in the example—and record those counts in a word-by-word matrix. An example pair might occur together five times. A compact embedding can then be imagined as retaining useful structure from that enormous table in far fewer columns. This compressed co-occurrence picture explains why embeddings contain contextual information; GPT-2 itself learns its token table jointly through next-token training rather than literally compressing that count matrix.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare directions, then look up the rows
Why compare embeddings using cosine similarity rather than straight-line distance? Cosine compares vector directions: aligned vectors score 1, orthogonal vectors score 0 and opposite directions score −1. Semantic opposites need not point in opposite directions. Happy and sad, for example, share emotional contexts. Anand reports that few pairwise similarities among GPT-2 token embeddings are negative, and those tend to lie near zero.
The co-occurrence analogy makes direction meaningful. Consider three words counted against the same two context words:
| Word | With word 10 | With word 11 | Ratio |
|---|---|---|---|
| word 1 | 10 | 10 | 1:1 |
| word 2 | 50 | 50 | 1:1 |
| word 3 | 5 | 20 | 1:4 |
The first two vectors, (10, 10) and (50, 50), have the same relative pattern, though the second word is five times as frequent. They point in the same direction. (5, 20) lies closer to (10, 10) in Euclidean distance but expresses a different contextual proportion. Comparing direction separates that proportion from raw frequency.
Using an embedding is much simpler than learning one. GPT-2 Small’s model_wte table has 50,257 rows and 768 columns. Each token ID selects one row. In the running prompt, the leading-space token is has ID 318; the spreadsheet-style interface displays that zero-indexed row as row 319. The matching decimals in the source table and prompt table demonstrate a lookup, not a new calculation.
javascript
function tokenEmbeddings(tokenIds, model_wte) {
return tokenIds.map(id => model_wte[id].slice());
}
The resulting matrix has one row per input token, preserving their order.
Embeddings also connect different modalities. CLIP learns relationships between images and text: repeated pairings of dog images with descriptions involving dogs can place the two representations into a useful relationship. The same general idea—learning comparable representations from associated observations—extends beyond word-to-word similarity.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Add where each token occurs
The dog chases the cat and The cat chases the dog contain the same words but describe different events. Token identity alone does not encode that difference. Some operations that combine numerical representations do not preserve order by themselves, so the model needs an explicit position signal. Anand illustrates this as offsetting a token such as woman according to where it appears in the prompt: related representations retain the token’s identity while carrying location information.
The original Transformer used sinusoidal position encodings. Sine and cosine oscillate between −1 and 1; the illustration represents that bounded variation as a small cloud around a token’s original position. GPT-2 instead learns a position table during training. The geometry is an intuition for adding position information, not a claim that each real embedding follows a literal two-dimensional circle.
GPT-2 Small’s model_wpe table has 1,024 position rows and 768 columns. Its rows correspond to the available positions in the model’s context. For each token, select its token vector and add the vector for its position, element by element:
javascript
function embedPrompt(tokenIds, model_wte, model_wpe) {
if (tokenIds.length > model_wpe.length) {
throw new RangeError("Prompt exceeds the position table");
}
return tokenIds.map((tokenId, position) =>
model_wte[tokenId].map(
(value, dimension) => value + model_wpe[position][dimension]
)
);
}
Only the position rows needed by the actual prompt are used. The vocabulary table is indexed by token ID; the position table is indexed by location.
A plot of happy at different positions shows the resulting offsets; labels such as happy 3 identify positions, not new vocabulary entries. Many newer models use RoPE, or rotary position embeddings, instead of GPT-2’s additive learned table. Asked how GPT-2 obtains its position values, Anand returns to training: make those values learnable parameters, then propagate the prediction error through them along with the other parameters.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Attention makes a token specific to its context
The representations now enter transformer blocks, commonly called layers. Anand uses “block” to avoid confusing a transformer layer with the layers inside its neural network. The implementation’s numbered steps preserve the layout of the Excel teaching material; attention occupies steps 4–9. The two main operations inside each block are multi-head attention and the multilayer perceptron.
Attention lets a token’s representation incorporate relevant information from other tokens. In the running sentence, He can use the earlier Mike to resolve its reference. Quick has several senses—physical speed, intelligence, the sensitive part of a fingernail, or alive in “the quick and the dead.” By the time the model reaches moves, the earlier quick helps make fast physical movement a useful interpretation for predicting the continuation.
Anand imagines tokens exerting a kind of gravity in embedding space: relevance determines which interactions matter, while values supply the information that changes the representation. In OpenAI’s GPT-2 implementation, the actual mechanism uses scaled query–key products, a causal mask, softmax and a weighted combination of value vectors. The metaphor’s useful consequence is contextual movement: the initial embedding for moves must accommodate both fast and slow uses, but attention to quick can shift the current hidden representation toward fast movement. It does not rewrite the stored vocabulary embedding.
Step 7 exposes an attention matrix, with prompt tokens on both axes. Each row describes how one position distributes attention over available positions. Further heads appear elsewhere in the wide table; Anand navigates to them by scrolling 64 spaces. The upper triangle is zero because GPT-2 is causal: a position can attend to itself and earlier positions, never later ones. Each row sums to one. In the displayed first head of the last block, moves assigns approximately 16% of its attention to Mike and 23% to is. Those values describe that particular head, block and prompt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A neural network is weighted arithmetic with activations
The second major operation in a block is the multilayer perceptron, or MLP. Artificial neural networks borrow inspiration from biological neurons without directly simulating a brain. The introductory analogy is activation: incoming signals may or may not cause a neuron to transmit a signal onward. The computational version multiplies inputs by weights, sums them, adds a bias and applies an activation function.
For one neuron, the operation is:
A simple activation is ReLU: negative inputs become zero, while positive inputs pass through unchanged.
javascript
function reluNeuron(inputs, weights, bias) {
const total = inputs.reduce(
(sum, value, index) => sum + value * weights[index],
bias
);
return Math.max(0, total);
}
Changing the weights and bias changes which inputs activate the neuron and how strongly.
An MLP arranges neurons into layers, with each neuron connected to all the outputs of the preceding layer. Layers between input and output are hidden layers. You will also encounter the terms fully connected network and feed-forward network; they overlap with MLP without being interchangeable in every context. Many weighted sums can be written together as matrix multiplication plus a bias, followed by activation. This is why the implementation repeatedly performs matrix multiply and add operations. Anand’s interactive matrix-multiplication example connects the compact matrix notation back to the individual neuron calculations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fit a function instead of writing it by hand
The attraction of an MLP is its ability to approximate functions. Start with a simple target, a parabola, and a network with one input, one output and two hidden ReLU neurons. One ReLU supplies a line segment for the right side; a flipped one supplies the left side. Adding them gives a rough piecewise-linear approximation over the displayed interval.
In the interactive demo, changing parameters moves those pieces while a mean squared error display tracks the mismatch. More neurons supply more pieces: Anand shows approximations using 8, 20 and 200 neurons. In the 200-neuron illustration, the approximation is visually difficult to distinguish from the parabola at the displayed scale. The universal approximation theorem motivates this capacity under suitable assumptions; it does not guarantee that a chosen training procedure will find the desired parameters.
For language modeling, the unknown function maps context to useful predictions. Training examples supply observed continuations, so parameters can be adjusted without a programmer writing the language rules or manually turning every knob. The distinction matters: backpropagation calculates gradients; an optimizer updates parameters using them. Anand’s foggy-mountain analogy describes the update process. The hiker’s position represents the parameters, elevation represents error, and the local slope suggests a direction that decreases error. The slope does not reveal the whole landscape or promise the best possible minimum. Anand also warns, from experience, that following downhill terrain is not a reliable real-world hiking strategy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Expand, activate, project—and repeat with new weights
In GPT-2 Small, the MLP maps a 768-dimensional input through a 3,072-dimensional hidden layer and back to 768 dimensions. The fourfold expansion is an empirical architectural choice. Steps 13–15 expose the three operations:
| Step | Operation | Output width |
|---|---|---|
| 13 | Multiply by expansion weights; add bias | 3,072 |
| 14 | Apply GELU element by element | 3,072 |
| 15 | Multiply by projection weights; add bias | 768 |
Unlike the introductory ReLU example, GPT-2 uses GELU. The same MLP is applied independently to each token position, normally in parallel. Its output updates a hidden representation through the block’s residual connection; it is not an independently decoded next token.
The code is a direct composition of the earlier arithmetic. With row-oriented input representations X, the expansion multiplies the previous step’s output by mlp_fc_weights and adds a learned bias. GELU transforms the result, then another learned weight matrix and bias project it back:
The two weight matrices perform different jobs and are learned separately.
Training reaches much further than the MLP. Gradients also flow through token embeddings, GPT-2’s learned position embeddings, attention’s query/key/value parameters and learned LayerNorm parameters. Anand compares the process to a chef reconstructing a dish: the ingredients are the input, the desired dish is the target, and the architecture prescribes available operations. Optimization determines the quantities that make those operations produce a useful result.
GPT-2 Small has 12 transformer blocks, each using the same kinds of operations with its own learned parameters. These successive transformations refine the hidden representations before final vocabulary scoring. In the parameter names, h0 denotes the first block and h11 the twelfth. The educational interface reuses its displayed block machinery by retrieving DOM formulas, replacing the block identifier and rerunning the steps with the appropriate weights. Repeated structure does not mean shared weights.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn the final representation into a token
After the last block, a final LayerNorm prepares the hidden representations for the language head. For next-token generation, use the representation at the final input position. It still contains 768 numbers, so it must be converted into scores over the vocabulary. GPT-2 reuses model_wte: each vocabulary row is dotted with that final vector. A 50,257-by-768 table multiplied by a 768-element column vector yields 50,257 scores, one per token. These scores are logits. They are dot-product compatibility scores, not cosine similarities or probabilities.
Softmax converts the logits into a probability distribution. Subtracting the largest score before exponentiation keeps the arithmetic numerically stable without changing the resulting probabilities:
javascript
function vocabularyLogits(finalHidden, model_wte) {
return model_wte.map(embedding =>
embedding.reduce(
(score, value, index) => score + value * finalHidden[index],
0
)
);
}
function softmax(logits) {
const maximum = logits.reduce((a, b) => Math.max(a, b), -Infinity);
const weights = logits.map(logit => Math.exp(logit - maximum));
const total = weights.reduce((sum, weight) => sum + weight, 0);
return weights.map(weight => weight / total);
}
The returned entries are nonnegative and sum to one.
The browser’s predicted-token cell takes a simpler path: it selects the maximum logit directly. Softmax preserves the ordering, so computing probabilities is unnecessary for this greedy choice. Anand uses deterministic decoding to make the output comparable with OpenAI’s code or Hugging Face Transformers. He submits the running prompt to the latter during the workshop, then leaves the request to finish while the network is slow.
Other decoding rules control which tokens can be sampled:
| Method | Selection rule |
|---|---|
| Greedy | Choose the highest-scoring token |
| Top-k | Keep a fixed number of highest-probability tokens |
| Top-p / nucleus | Keep the smallest leading set reaching a probability threshold |
Anand illustrates top-k with ten candidates and top-p with thresholds such as 80% or 90%. After restricting the candidates, renormalize their probabilities before sampling. These rules reduce the chance of drawing an extremely unlikely continuation such as haircut.
When the delayed Hugging Face result returns, it agrees with the browser example: Mike is quick. He moves continues with quickly. The displayed winning token is ID 2952, with a maximum logit of approximately −129.44. The spreadsheet shows it on row 2953 because its visible row numbering starts at one. A negative score can win: the decision depends on being larger than the other logits, not on being positive. Looking up ID 2952 in the token dictionary completes the return from numbers to text.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Internet completion is not assistant behavior
The move from GPT-2 to ChatGPT involved scale and architectural changes, but a central distinction is the behavior training asks the model to produce. A base model learns to continue internet text. Given First name:, GPT-2 produces form-like continuations involving passwords and email. Given Hello class, it heads toward code containing Foo and public static void main, rather than replying as a student or teacher. Those continuations make sense as text patterns even when they are unhelpful as conversation.
The same model can answer that France’s capital is Paris, so useful knowledge is mixed with other continuation patterns. Assistant training tries to make the desired behavior more dependable. Anand introduces a four-stage progression, starting with internet pretraining and then supervised training on examples of helpful responses. He describes a general range of 10,000–100,000 contractor-written prompt/response examples for that assistant-training stage.
The Stanford Alpaca dataset makes the instruction/response format easy to inspect as JSON. Anand shows examples asking for healthy-living tips and primary colors. Alpaca is a format illustration here: its outputs were generated by text-davinci-003, rather than being the contractor-written examples just described. Supervised fine-tuning teaches the model to imitate this kind of desired response, but imitation alone does not capture every preference about helpfulness and safety. That motivates reinforcement learning from human feedback.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn a score, then optimize the assistant against it
Imagine an agent navigating a maze with monsters, obstacles, power-ups and a goal. It tries routes, receives scores and learns a policy that favors better outcomes. Unlike copying a demonstrated route, reinforcement learning can search for behavior based on its reward. A generated sentence can likewise be viewed as a path through successive token choices. But scoring language is more subtle than counting desirable or undesirable words: not angry cannot be judged by treating angry in isolation.
To obtain a scoring function, collect preferences between responses to the same prompt. Anand shows examples from Anthropic’s helpful and harmless preference data. For a pumpkin-pie request, the preferred response supplies a recipe, while the rejected one tells the user to buy pumpkin and read the package. A harmlessness example prefers constructive guidance about stress-related drinking over encouragement to drink without restraint. Such pairs express judgments about the response as a whole.
The distinction between the last two stages is crucial:
- Pretraining: learn broad text patterns from internet passages.
- Supervised fine-tuning: imitate examples of desired assistant responses.
- Reward modeling: use chosen/rejected response pairs to train a separate scoring model.
- Reinforcement learning: update the assistant to favor responses that score well under that reward model.
At stage three, learning the scorer has not yet updated the assistant itself. Stage four uses that scorer to change the assistant’s behavior. Anand closes this part by pointing to R1-Zero and GRPO as developments making reinforcement learning especially relevant beyond the earlier assistant-training recipe.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A recommendation system for the next token
The whole pipeline can also be understood through recommendation systems. Tokenization finds an efficient representation of the input. Embeddings then capture relationships, much as representations of books, films or music can group related items from user behavior. Anand shows music-genre clustering as an example. A word’s contextual relationships provide useful clues about plausible continuations, just as an item’s relationships help recommend another item.
The neural network learns to turn those clues into predictions. Attention adds the current passage’s evidence: moves in the presence of an earlier quick should support different continuations from moves in a slow context. Successive blocks refine that representation, and the language head scores the vocabulary. The goal of walking through the implementation is not instant mastery of every operation. It is recognizing that each operation is understandable, inspectable machinery.
For continued study, Anand directs attendees to the workshop’s Discord survey and mailing-list signup for the PDFs, along with the project’s mailing list and YouTube channel. He also mentions Patreon, consulting and training. The closing questions then return to two practical consequences of this foundation: how to evaluate prompts and how newer models increase capacity without activating every parameter.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Dictate freely, but test engineered prompts
An audience member describes using superwhisper on a Mac to dictate long prompts, ramble through relevant information and repeat important points. Is that a good approach? Anand’s first answer is empirical: theories about model behavior need tests. He cites The Prompt Report and work by Ethan Mollick’s team while discussing how the usefulness of politeness and emotional appeals can change across models. His practical lesson is to avoid treating a remembered prompting trick as a permanent rule.
For his own one-off use, Anand also dictates a brain dump, then corrects mistakes, repetition and awkward grammar. His intuition is that a fixed forward pass has bounded computation: making the model decipher noisy input may make the requested task harder. That is a heuristic about input quality, not a measured exchange rate between spelling errors and reasoning capacity. Reasoning models qualify the picture because they can use additional generated steps. Relevant information is still more valuable than omitting useful context merely to make the prompt cleaner.
For an engineered prompt, establish an evaluation baseline and measure changes against it. Keep representative tasks, compare prompt variants and rerun the evaluation when the model changes. Understanding the architecture suggests hypotheses; evals tell you whether those hypotheses improve the behavior you actually need.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More parameters without using all of them for every token
The final question asks about increasingly fine-grained mixture-of-experts models, mentioning Llama. Anand places MoE alongside RoPE, RLHF and reasoning models as major extensions to investigate after learning the GPT-2 foundation. In the architecture he describes, MoE changes the MLP portion of the transformer, which accounts for a substantial share of its computation.
Instead of sending every token through all the available feed-forward capacity, route each token to a subset of experts. The model can then have more total parameters while activating only part of them for a particular token. Expert weights can also be distributed across devices. That deployment arrangement can help manage memory, but inactive experts do not cease to require storage. The distinction is between total parameter capacity and the computation active on each token.
The difficulty shifts toward training and coordination. Anand identifies training complexity as one reason open implementations took time to mature, and notes that MoE predates ChatGPT. Its purpose is a concrete extension of the machinery already examined: retain more learned capacity while controlling how much of it must run for each prediction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explore GPT-2 through JavaScript cells and intermediate tables. Desktop Chrome and locally loaded model CSV files are required.
The Excel-based GPT-2 project underlying the workshop's teaching approach.
Historical implementation of embeddings, causal attention, MLP blocks, residual connections and vocabulary scoring.
52K generated instruction-following examples with instruction, optional input and output fields; subject to research and noncommercial restrictions.
Human preference data accompanying Anthropic's research on helpful and harmless assistants.
A systematic survey organizing prompting terminology, techniques and research.
Further reading
Sennrich, Haddow and Birch's 2016 paper on subword representations for rare and unseen words.
- ArrrZeroArticle
Anand's companion discussion of DeepSeek R1-Zero and reinforcement learning.
Updates since the talk
An August 2025 study of tipping and threatening prompts on GPQA and MMLU-Pro, reporting aggregate and question-level effects.
Read the complete timestamped transcript
- 0:00
[on-hold music] Okay.
- 0:17
Thank you for coming bright and early, uh, nine AM at the start of the conference, uh, for LLMs for web devs, uh, GPT in six hundred lines of vanilla JavaScript.
- 0:31
Um, I think you [REDACTED:gender] are gonna have a great conference. I was here last year. I thoroughly enjoyed it, um, and I think this is a great way to kick things off.
- 0:39
If you're just coming to this field or this conference without any background in machine learning, this is your missing AI degree that'll help make the rest of the conference, I think, hopefully a lot more valuable.
- 0:52
So if you're just joining us and came in, you can go to spreadsheets-are-all-you-need.ai, um, and there's a Discord link in the upper menu. Uh, click on that, and then go to the AI Engineer World's Fair twenty twenty-five room, and then there's a link to download the GPT-2 model weights in the pinned message at the top.
- 1:13
Takes a little while to download that, so I would download that and get started, uh, if... 'cause the Wi-Fi might be a little bit slow. We're gonna be using that in a bit.
- 1:23
Okay. We're gonna do something special today. Uh, it is a talk, and I will be doing a lot of talking at you, and we'll be running code, so it is a workshop, but our mission today is to break Clarke's third law.
- 1:40
You might be familiar with the science fiction author Arthur C. Clarke and his famous maxim that any sufficiently advanced technology is indistinguishable from magic. And nowhere is that, you know, more true or relevant today than when it comes to large language models.
- 1:56
These are seemingly magical machines that can produce lifelike text, automate tasks as agents, and maybe even replace humans in certain contexts.
- 2:08
And if you ask somebody how these work or you go online, you're liable to get the impression that you need to have semesters of linear algebra and calculus before you can begin taking your first machine learning class and understand how these work.
- 2:24
And yes, that's true if you wanna be a machine learning engineer, but if you just wanna understand how these work and you're a builder on top of them, I'm here to tell you that is not true.
- 2:35
Do not believe them. You don't need all that sophistication if you just wanna have a really accurate model of how a transformer works. I know because I was here last year.
- 2:47
I gave a talk called Spreadsheets Are All You Need, where I showed an Excel worksheet that implemented all of GPT-2 Small entirely in pure Excel functions. And then I took that spreadsheet and I turned it into a class that I taught online, where I took people tab by tab through how the entire model works, and not everyone
- 3:07
was even an engineer. So one of my favorites is this [REDACTED:gender]. This Joe here is a CFO. He's just naturally good ex- at Excel, and that gave him everything he needed, combined with the sheet, to understand how a large language model works on the inside.
- 3:22
And he says, "This is great. I, I had no experience with machine learning or AI concepts before." Yet he was able to walk away with a very good understanding of how they work.
- 3:32
So what I'm gonna do today is compress that class, which is about eight hours, down to these two hours today, and explain how the transformer works. And I show up today or this year at the conference with this.
- 3:45
Instead of Excel, because not every one of us have a job that require us to be really good at Excel, uh, we're using a vanilla JavaScript implementation because a lot of folks who come to AI engineering as a field have a web development or full stack JavaScript background.
- 4:02
And so if that's your background, you are perfect the way you are. You don't need to learn Python if you just wanna understand how a model works, and you're still gonna use TypeScript and Next.js around the model, but you still wanna have a good understanding of how it works.
- 4:17
So today's approach is I'm gonna give you the background to understand the code, and we're gonna take a, a brief walk-through of it. I'm gonna focus more on the why than on the what.
- 4:29
So we'll take a look at the code, but I'm gonna spend a lot of time building intuition and background to understand the code.
- 4:36
And then instead of complex equations, I'm gonna use analogies and examples to make it more tangible.
- 4:44
Okay. And the background you need to have is, first of all, just motivation, uh, and a curiosity to understand how these work on the inside. Uh, any science, technology, engineering background is sufficient.
- 4:56
Prior programming experience, especially in JavaScript, uh, but you don't need to know React or Vue. We're gonna just use vanilla JavaScript. And then some awareness, I would call it, of linear algebra, meaning you just need to know what a matrix multiplication is.
- 5:10
Uh, you don't need to be a hotshot JavaScript ninja, you don't need prior AI or ML background, and you don't need, you know, deep calculus or linear algebra fluency.
- 5:22
Okay, and the key resources for today are going to be our JavaScript implementation of GPT-2. Uh, use Chrome on the desktop, and if you go to spreadsheets-are-all-you-need.ai, uh, and I, I apologize, that is a long domain name, slash gpt2, it'll load up this implementation, and it'll run locally in your browser.
- 5:41
Uh, there's a Discord server where I've dropped some links, and if you've got questions, feel free to drop them in there as well.
- 5:49
Okay. So this is a simplified diagram of GPT-2. It does not look like your classic transformer diagram intentionally, and it will serve as our roadmap for what we're gonna do throughout the, today's workshop.
- 6:04
I'm gonna start by just giving you some background on LLMs and our JavaScript implementation of GPT-2, how to get it running, how to get it started. And then we're gonna focus on these three areas for the most of it, and that's tokenization, embeddings, and then the language head.
- 6:22
And I've focused on those because that's the input and the output of the model, and those are the most important to have the background and understanding if you're going to be building systems around and on top of LLMs.
- 6:34
We will cover the inside number-crunching part, that's attention and the multilayer perceptron at a pretty high level. I'll go a little bit more into the multilayer perceptron because then I can explain about backpropagation, and it serves as kind of a foundation to understand how the model learns, uh, which is an important concept.
- 6:55
And then finally, I'll talk about the difference between GPT-2 and ChatGPT. So they were separated by three or four years. What were those innovations that made the model so much seemingly more smarter than GPT-2?
- 7:09
Hint, it wasn't necessarily, uh, anything algorithmic. Okay. So let's start with a quick tour of our JavaScript implementation of GPT-2 and then some background on LLMs.
- 7:23
So this is what you get when you load up GPT-2, uh, slash GPT-2 on the Spreadsheets-are-all-you-need website. Uh, if you scroll down, the first thing you're gonna wanna do, there's a link here, says, "Download the GPT-2 small CSV."
- 7:42
So the first thing you wanna do is go to this page on GitHub. This is all the model parameters of GPT-2 small in a bunch of CSV files. It's a giant zip file.
- 7:52
Uh, you're gonna download it and unzip it. And when you hear about, you know, this model is like a billion parameters or seventy billion parameters, that-- it's all just a bunch of giant numbers, and you can think of it as a giant spreadsheet, and that's literally what the zip file is.
- 8:08
Once you've downloaded that zip file and opened it up, what you wanna do is select all of those files and drag it into this section here. You can-- Once, uh, when it, uh, when it's done loading all those files, it'll look like this.
- 8:23
It'll say, "Ready: All model parameters loaded." When it's not loaded, it'll be in red, and it'll say, you know, "Please add the files." And what it's doing is it's actually loading all those GPT-2 parameters locally into your indexed DB database.
- 8:37
And you can see here it's basically about one point five gigabytes. Now, Chrome will let you do that, uh, if you've got sufficient disk space on your hard drive.
- 8:46
But the benefit of doing this is now the entire model is running locally in Vanilla JavaScript on your browser. In fact, to run and debug this, you don't need anything else.
- 8:57
You could pull the internet connection and it should still work. And the way this is set up is similar to a Python notebook, if you've encountered one of those.
- 9:06
We have these cells, uh, and every cell has a play button which will run what's inside it, and there are two types of cells. One type of cell is just JavaScript code, and you can open these and expand these if you want.
- 9:18
It's just Vanilla JavaScript code. Our matrices, for example, are just simple two D JavaScript arrays. And if I hit play, it'll execute this code, and you can see there's a, a message right there.
- 9:28
The other type of cell is one like this. It's kind of like a table or spreadsheet interface. It runs a formula and then shows you the result. So it's a way to actually run code and see the results very immediately.
- 9:43
And these formulas are just raw JavaScript with a little syntactic sugar to figure out, you know, if you reference something, it knows what previous table it was.
- 9:53
Let me give you an example of that.
- 9:56
So right here is an example. So here, this getFinalTokens is just JavaScript code we defined earlier, and this promptToTokens is literally the DOM ID and this-- Let's zoom this one out.
- 10:11
This, uh, this brackets is basically syntactic sugar saying, "Go grab the DOM table that has this ID." So promptToTokens is up here, and it's just grabbing this thing and passing that into that function.
- 10:25
So, uh, it's all straight Vanilla JavaScript, but the real benefit is being able to debug a model right here without leaving your browser. So what I'm gonna do, I can do...
- 10:38
Who here has done like console debugging before, right?
- 10:43
Right. So you can do console debugging right here. So if I say console.log matches and I open up my dev tools inspector.
- 10:53
Let's put that side by side. There we go. And I go to the console, and then I rerun this.
- 11:03
There we go. Wait for the layout shifts.
- 11:09
There we go. Separate into words is right there. So if I rerun this, you can see right here, it might be a little bit hard, but if you, uh, look here, you can see I've basically got my console.log statement here.
- 11:23
If I wanted to know what that variable is doing, but I can go even further. I can just type the word debugger,
- 11:32
and then I rerun it, and boom. I'm actually stepping through a large language model that was once considered too dangerous to release right with ever leaving my browser. So every part of this model, if you were like, "I really wanna understand how this works," you can step right through it in a familiar language and the browser, a
- 11:50
very familiar IDE. So I'm gonna remove that debugger statement
- 11:56
so it doesn't get in our way later.
- 12:00
And then Okay, so that's a quick tour of how to run our JavaScript implementation of GPT-2.
- 12:18
Okay, uh, next up, large language models. Um, so a large language model has a really simple job to do. We give it a passage of text, and it simply predicts the next word, technically the next token, as we'll talk about.
- 12:33
So I might give it the text, "Mike is quick. He moves," and it'll just output a single word, quickly. It does not, by nature, nat-naturally give you paragraphs of text.
- 12:44
So if we wanna get more text, what we do is we take that output, and then we append the word we just got out, which was quickly, and put it at the end of the thing w-we originally put in.
- 12:55
So then we take that additional text. Now we ask it to run through again with this much longer piece and say, "What is the next word now?" And it says, "Oh, it's and."
- 13:03
Then we take that, append it to the original input and keep going and ask it what the next thing is. And this is how we generate paragraphs of text or code from a model.
- 13:11
It is what they call an autoregressive model. You simply take the output, put it back to the input, and rerun it. Now, this is why if you use our JavaScript implementation, all it does is predict the next word because that is the core function.
- 13:24
If you understand that, you understand how the rest of it works.
- 13:28
So we've said that large language models have this core action of completing passages of text, and they're trained to complete sentences like this one: "Mike is quick. He moves..."
- 13:39
And as a human, you probably understand, you know, a possible completion is the word quickly or maybe the word fast or around. But how do we get a computer to do that?
- 13:48
Well, here's a fill-in-the-blank problem that computers are really good at. Two plus two equals four. It's a math problem. Computers are really good at math. And you can make these equations really complex, and computers can still do them really fast.
- 14:02
So in effect, what researchers have figured out how to do is take what is a word problem and turn it into a math problem. In order to do that, they have to go through a series of steps.
- 14:13
First, what they have to do is they have to map the words in our text to numbers. Here, I've shown it as just a one-to-one mapping, so Mike goes to eighty-nine, is goes to nine.
- 14:23
But in practice, as we'll see, it's a long list of numbers called an embedding,
- 14:28
and then we do our number crunching on them. Here, I've drawn this as just simple arithmetic. It's much more complex than that, but it's actually almost as simple as that.
- 14:36
It's just a lot of multiplication, addition. There's an exponentiation in there, um, but it's not math you probably haven't seen before. It's just a lot of it tediously put together.
- 14:49
And then after all that arithmetic, we get a result. Again, it'll be a long list of numbers. Here, I've simplified for now, just saying a single number, and we look at the resulting number that comes back, and that number is going to be what it says the next predicted word is going to be.
- 15:05
But we need to translate that back to a word because it's a number. So then we do the reverse of what we did at the beginning. Instead of going from words to numbers, we go from numbers to words.
- 15:13
And of course, the number we get back, numbers are continuous, uh, words are discrete, doesn't necessarily always cleanly map. We'll get some number like here, for example, hypothetically, we get two thirty-one.
- 15:24
There's nothing in our dictionary that maps to it, but the closest word in our dictionary is quickly, which is at two thirty-two. But fast is kinda close too at two forty.
- 15:33
So what we're going to do is we're gonna weight the probability distribution of these tokens according to how close they are to the predicted number that came out of our model, and that turns into our probability distribution.
- 15:46
So then we run a random number generator, and then we pick according to that distribution. One thing I wanna emphasize is we add the random number generator in. We could always just simply take the closest word, and that's called greedy or temperature zero.
- 16:02
Okay, so that gives us this view. You get some text. We turn that text into tokens. We turn those tokens into numbers, and then we do some number crunching on them, and then we turn those numbers into text,
- 16:15
and that gives us our next predicted token.
- 16:19
Okay, the model we are going to be studying today is GPT-2, uh, GPT-2 small specifically. There's actually multiple versions of GPT-2 that were released, uh, and that came out in twenty nineteen, so about four years before GPT-4 and three years before ChatGPT.
- 16:36
But don't let that fool you. This was a model that was considered, uh, too dangerous to release when it first came out and was state-of-the-art. More importantly, it is actually the foundation of most of the state-of-the-art models you have probably used today.
- 16:51
And you don't have to take my word for it. This is a research lab, EleutherAI, saying basically, the recipe for building a large language model has not fundamentally changed since the Transformer was introduced and only slightly tweaked from the language models by OpenAI, GPT-1 and GPT-2.
- 17:08
And then in this article, they actually go on to list what the changes are between GPT-2 and, uh, so, and a state-of-the-art model at the time, which was Llama 2, when this was written, which was last year.
- 17:21
So the way to think about this, and this is a helpful family tree chart of different large language models, is that most of the ones you're familiar with, at the top of this tree, might be hard to see, is ChatGPT, Llama, Bard/Gemini, GPT-4, Claude.
- 17:36
They all inherit from GPT-2. GPT-2 is its granddaddy. If you understand GPT-2, you are eighty percent of the way to understanding how a state-of-the-art model works under the hood.
- 17:50
Okay, so now let's dive into the first stage of our model. So that's tokenization.
- 17:58
Okay, this is where we take the input text, and we split it into subword units called tokens. In the example that I like to use, Mike is quick, he moves, unfortunately, every single word is a single token.
- 18:11
But it is not unusual For a word to be two, three or more tokens. And then these tokens all have IDs that are just lists or positions in the dictionary, as you see here underneath.
- 18:22
So let me show you what this looks like.
- 18:26
So I'll wait for the Wi-Fi. Okay, well, we get more people in the room, we get, we get
- 18:51
more Wi-Fi, uh, issues. Um, so what I want to illustrate for you is that you can take a word like-- Uh, oh, you know what I can do? I, I do have a backup.
- 19:03
Let's do this. Okay, this is the version that runs locally.
- 19:11
Uh, so what you can do-- So you can try this once we get Wi-Fi back. I'm gonna take the word reindeer and the word re-injury, and then I'm gonna run them up until right here, which is the final tokens.
- 19:30
So these are the tokens for the input prompt we just put in here.
- 19:36
And you can see the word re-injury was turned into multiple tokens. There's a space, which is part of the token itself, R-E-I-N, and then jury, J-U-R-Y, and they have separate token IDs.
- 19:49
The thing I want you to pay attention to is reindeer also starts as R-E-I-N, right? But it got split into three tokens: space, R-E-I-N-D-E-E-R. So this is not like basic string parsing.
- 20:03
Something more complex is going on. And so the natural question is, well, why the heck are we doing this?
- 20:09
Why don't we do something simpler? Why don't we do, say, word-based tokenization? We just take every word in the dictionary and give it a number. Like dog is one, cat is two, and so forth.
- 20:19
So that has a couple of problems. First is it can't handle unknown or misspelled words. And there are some models, early models, that had an UNK for unknown token.
- 20:27
Um, but when you're grabbing all the text on the internet, you might encounter things you didn't expect. Examples could be languages that you weren't planning for. One of the early models, GPT-2, in fact, they tried to take foreign languages out of it, and then they magically discovered some snuck in, and it was actually good at translation.
- 20:44
They wouldn't have had that if a lot of words were just simply-- that were not English were just thrown out. Another example is when they did summarization with it, they realized they can put the too long, didn't read acronym TL;DR, and if they didn't have a token for that, it would have been thrown away, and it would
- 20:59
have lost that ability. The other problem is that you're gonna increase the vocabulary size, which is going to increase the size of the model. It'll need more parameters if you're going to have more vocabulary.
- 21:11
English alone is a hundred and seventy thousand words. For perspective, GPT-2's vocabulary is only about fifty thousand, so it's a third of that. And then if you add additional languages on that and you're doing word-based tokenization, it would get even larger.
- 21:23
So in essence, if you do this, you get more memory, more compute, or maybe less performance. So then you're like, "Well, I'm a developer. I'm used to say something like ASCII.
- 21:32
Why don't I just do character-based tokenization?" I say A is one, B is two, and do it that way. Well, the first problem is it's going to increase the sequence length.
- 21:42
So we can see this inside the model.
- 21:46
As you go through the model, after you get your prompt, right here, you can see here is where the embeddings, we'll talk about those in a second. But you can see Mike is quick, period, he moves.
- 21:56
Each of these rows, this matrix, has a height that is the size of the number of tokens, right? And that persists through the entire model. So as I keep going, we see again this six-height matrix, it's gonna keep going.
- 22:09
If we made every single character its own token, this is gonna get a lot larger. Right now, it's just what? Six tokens high. But if I made M-I-K-E, each of these characters their own token, this is gonna get a much larger matrix, so it'll be more memory, more compute to process.
- 22:26
The other issue is there's low semantic correlation in characters. They don't carry a lot of meaning. And a good example was this chain letter that went around a few decades ago on the Internet.
- 22:35
And it says, "According to research at Cambridge University, it doesn't matter in what order the letters in a word are. The only important thing is that the first and last letter be at the right place."
- 22:44
And all the letters are jumbled, but you can still read it. And the point is that you don't read characters, you actually read sub-word units yourself. Um, and so if there's less semantic correlation, it's gonna be more work for the model to do during training to erase that character boundary and get the pieces that really matter.
- 23:02
So if character tokenization is too small and word tokenization is too big, Goldilocks says, "Let's do something in between," which is sub-word tokenization, and that's this algorithm called byte-pair encoding.
- 23:13
So it's got two phases. The first is the learning phase, where you take a large, they call it a corpus of text that's gathered from the Internet, and then we put it through this learning algorithm we'll describe in a second, and then we get out of it a vocabulary, a dictionary of tokens.
- 23:28
And then later, when we're processing the model and asking it to generate text, if we give it some input words, we have to retranslate it into those tokens that were used during translation-- during training.
- 23:39
So we take the input words, we take the vocabulary, and we get out tokens.
- 23:44
This is the research paper that introduced the algorithm to machine learning, but it turns out this algorithm is from the nineties. It's actually a compression algorithm, as we'll talk about in a second.
- 23:55
And it even has some Python code you can copy and paste and run. The goal of the algorithm really is to take the text that's going to be trained on and figure out the most efficient way to represent it.
- 24:06
That's really what tokenization is trying to do. And so here's the example from the paper. And what I'm gonna do first is I'm gonna use this dot To separate out the characters, right?
- 24:17
That's gonna tell us essentially each token will be separated by those dots so we can see them individually. At the end of this process, you'll see that there are less dots, meaning that we've got more tokens-- Sorry, a lar- more number of tokens in our vocabulary, but less number of tokens being used to represent the corpus.
- 24:34
And in this corpus, we're gonna pretend that when we scraped the internet, this is all we came up with. There are only four words: low, lower, newest, and widest.
- 24:43
And you'll notice some of them appear more than once. In fact, they all do. But the reason I'm doing it this way is I want the frequency of the word to be represented, right?
- 24:50
I'm trying to compress all the, the wor-words on the internet, and if a word is more frequent, I wanna know that. Because if it's more frequent, I wanna give it more, more representation or a more efficient representation.
- 25:03
So that dot is gonna separate out our tokens, and then I'm gonna put the underscore to indicate the space character. One caveat, in this example, the space character is at the end of the token.
- 25:13
In GPT-2, it's actually at the beginning. And then we're gonna start with our vocabulary of just our characters, A through Z, all lowercase. Uh, we're assuming that we're in a lowercase only world here for now.
- 25:24
Those are our initial tokens. That's our vocabulary on the right. And then the first thing we're gonna do is we're going to count all the adjacent tokens, all the adjacent characters.
- 25:34
Right now, it's-- there are no tokens other than characters. And we can see E and S occurs nine times, right? Six times in newest and three times in widest.
- 25:44
So then I'm gonna put a table together, and I put E next to S has a frequency of nine. And then I'm gonna do this for all the possible pairs.
- 25:50
And then I'm gonna take the most frequent pair, and I'm gonna say that's a new token. Why am I doing that? Because if I do that, I can take E and S, I can put them together, and I can pretend they're their own character.
- 26:02
I'll call it a token. And then I can go back to my corpus, and every place there is an E and S, I'm gonna replace it with my new ES token.
- 26:10
So now I've shrunk the number of tokens needed to represent the stuff here on the left. Now, I've taken what were separate tokens and combined them together, so I'm using less tokens to represent all this text.
- 26:21
And then I can repeat the process. I can just say ES next to T occurs nine times, and I can do a whole 'nother table. Now, ES is itself now its own character, its own token, so it can be paired with other things that I'm counting.
- 26:35
And I can see that ES with T occurs nine times. So I add that to my vocabulary of tokens, and I go back and I compress my corpus again.
- 26:44
Then I keep going, and I just simply repeat this process, looking for whatever is the most frequent at each pass and making it a token and then recompressing the corpus.
- 26:53
And after ten passes, you get something like this. Here on the right is our vocabulary of tokens. And then here on the left, you can see we have shrunk the number of tokens used to represent the corpus.
- 27:05
There are less of these dots, right, than you-- we saw before and originally. Now, the most frequent words became their own tokens, right? Low and newest. And even the words that did not get their own full token representation, we've now represented them a lot more efficiently.
- 27:25
Uh, and you notice that there's some common subword units, like low became tokens and EST became tokens. So it managed to map to some of the morphemes, the parts of speech we use as humans as tokens, but that is just a coincidence.
- 27:40
Right now, the model has no understanding of semantic meaning. Okay, so this is the learning algorithm. Now, the tokenization algorithm is really similar. I'm not gonna go through it in full detail.
- 27:51
I have a video on my YouTube channel where I do go through it in full detail, but for time, I'm just gonna talk about it at a high level.
- 27:59
It's essentially similar to the learning algorithm, except we're doing it on individual tokens as they come in, asking ourselves, "What would it look like if this word were part of the tokenization process?"
- 28:10
So let me show you what that is like. So there's a helpful jump to here. Let's go to tokenization. So the first thing we do is we take our prompt, and we separate it into words.
- 28:19
Now, this is just a regular expression that came from the OpenAI source code when they open-sourced it, and we're just parsing it according to this, and it's gonna basically take out punctuation and spaces, so we get these as our words.
- 28:30
So Mike is quick, period gets its own token, he, and moves. Right now, we're not tokens, we're just separating to words. But do note that the spaces are assigned as part of this separation, so he and moves have a space in front of them.
- 28:47
The next thing we're gonna do is we're gonna fetch vocab BPE. This is a file that came from OpenAI when they trained the model on its tokenization. This is their dictionary of tokens.
- 28:55
So here's... When they did their training in BP, the most common left token with a right token was space with a T. The second most was space followed by an A, and so forth.
- 29:07
I've added two extra columns, rank and score, just to figure out where, uh, a pair of tokens is in terms of relative to the others, how important they were.
- 29:17
Then this is just a helpful map of tokens to their IDs.
- 29:22
And then finally, there's this, which is prompt to tokens. Now, this is not how you'd really wanna write a tokenizer. I've set it up to look similar to the Excel version of the matrix, so you can watch the video and still be able to watch it and, and understand it depend-- no matter which version of, uh, the
- 29:38
Excel sheet or the JavaScript version you're looking at. But this kind of illustrates the process. So here is quick. We're breaking it apart right here. Let me get Presentify.
- 29:50
There we go. We're breaking it apart into characters here,
- 29:53
right? Quick. And then what we're doing is we're looking at each pair of characters. So Q with U, U with I, I with C, C with K, and we're saying, for each one, where does it fall in that rank of tokens?
- 30:07
And we're gonna take the most popular one. In this case, it's I and C. And so we've rewritten this as a string of tokens, except the I and C were put together as one token.
- 30:17
And then we're simply gonna repeat the process
- 30:22
And keep going and we combine. Then Q and U get combined, then I, C, K get combined, and finally gets to the point where it realizes, oh, this is in my token vocabulary, and it turns it into a token.
- 30:34
Again, I have a video if you wanna see this in, in depth and in detail. Um,
- 30:40
but for now, just think of it as the same part of the learning process, really just run small. Now, tokenization is actually kind of considered by a lot of experts to be a necessary evil.
- 30:52
Some of the problems that you sometimes encounter with models, although the root cause isn't tokenization, it can sometimes make them worse. So the common one is how many Rs are there in strawberry?
- 31:03
Uh, models don't actually see the Rs. And I love this, you know, post from Riley Goodside, where if you remember The Matrix, the character says, "Oh, I don't, I don't see the numbers.
- 31:14
I just see, you know, what's inside there." And that's kind of like what it is to the model. You know, it doesn't see any of the letters. So what he's done here in this image is strawberry is tokenized multiple different ways, depending on whether it's uppercase versus lowercase, whether there's a space in front of it or not,
- 31:31
and whether there's a, a quote in front of it or not. So to the model, you know, these six words are not all strawberry. They're all six different token patterns.
- 31:41
And so it makes it a lot harder and a lot more work for the model to understand it. If you think about it, you don't actually see those letters either.
- 31:47
If somebody asked you how many letters are in a word, you'd have to stop and think and parse it out.
- 31:53
Um, I said at the beginning, you know, you don't wanna do word tokenization, you don't wanna do character tokenization. That's not a hard and fast rule. That's an empirical rule.
- 32:01
And there are research models that have done both. Uh, so just know that, you know, that's, you know, maybe a few years from now, there'll be character-based tokenization will be more popular.
- 32:10
This is an example of one. The last thing I wanna leave you with is that tokenization doesn't have to be just about, uh, text. It can also be about other things.
- 32:20
So here's an example of Vision Transformer. They use patches of images as tokens. Waymo uses, uh, trajectories in space to prevent collisions as tokens.
- 32:32
Okay. Let's briefly check how we're doing if this will load.
- 32:48
There we go. So this is actually pinned at the top. Uh, nine thirty-- Oh, we're right-- We're just a minute behind. Okay. So this is... You [REDACTED:gender] can keep me honest.
- 32:57
Uh, at-- In the Discord room at the top is a spreadsheet. I'm, of course, using a spreadsheet to make sure we stay on track and on time. Uh, okay.
- 33:06
So... Oh, we need to do this now.
- 33:12
There we go. Next up is embeddings. So now we're in the second phase of the input. Uh, these are, uh, token and position embeddings.
- 33:21
Okay. So at the beginning of this workshop, I talked about how we map words into numbers, and I simplified this process by saying what we're doing is we're taking, say, the word Mike, and we're turning it into a single number, uh, like eighty-nine.
- 33:36
But of course, that's not what we're really doing. We're actually gonna turn it into a long list of numbers called an embedding. Um, so even the period, for example, gets seven hundred and sixty-eight of these numbers.
- 33:47
And in the case of GPT-2, uh, the dimensionality, that list size is seven hundred and sixty-eight. Every single word gets seven hundred and sixty-eight numbers, and you can see that if we go to
- 34:00
the token embedding section. And then these... We'll get to that table in a second. These are the embeddings of our input prompt. And you can type... By the way, I, I didn't mention this before.
- 34:09
You can type anything here in this input prompt, and it should parse it, although it doesn't handle foreign language as well because there's a character mismatch on import. Um, but here's Mike is quick, period, and each of these gets seven hundred and sixty-eight numbers.
- 34:20
So what you can do is if I take row one, column seven sixty-eight,
- 34:28
there's where our list of seven hundred and sixty-eight numbers ends. So every single one of these gets the same number of, uh, dimensions for how it gets represented.
- 34:40
And it might be a little confusing 'cause we mapped words into tokens with numbers. We've had token IDs, and then we have these embeddings as well. And the analogy I like to use is imagine you are going to go, uh, look for a house to rent or a house to buy.
- 34:53
And the street addresses, uh, and you build this table of street addresses and square feet, bedrooms and bathrooms and price. The street addresses are identifiers. They're kind of like the token IDs.
- 35:04
They tell you where to find something, where to find a house, but they don't tell you anything about what's inside. They don't tell you about what you care about, what the meaning is.
- 35:11
It doesn't tell you the square feet, the bedrooms, the bathrooms, or the price. And the embedding values, that's their job. They're to take the token and tell you something about what it means.
- 35:19
And the identifier or the token ID is just to tell-- give it a position in the dictionary, effectively a numerical name.
- 35:26
And what we're really doing with the embedding values is we're trying to build a map for words where we put similar words grouped together. So here I've shown a two-dimensional map, but of course, we're in a seven hundred and sixty-eight hyperdimensional area.
- 35:42
But the idea is basically the same. Take the words happy and glad in this map, this word island. You know, those are happy words, so I'm gonna put them up here.
- 35:51
And then I'm gonna take words like dog and cat, and I'll put them in another part of this word island because they're animals. They, they don't relate as much.
- 35:59
And then, you know, the word sad, well, it doesn't have the same meaning as happy and glad, but it's still an emotion. So I want it closer to the emotions, happy emotions part of the island into this maybe the sad province right next to the happy province.
- 36:14
So they're kind of on the same half of the island, but they're not directly in the same spot because happy and sad have different meanings.
- 36:22
So here's a simple two-dimensional example of the benefit of doing this, which is that you can start doing word arithmetic and word math. So we're gonna imagine that we've built a two-dimensional embedding where the first column is authority and the second column is gender, and we take the token man, and we'll just arbitrarily say man has authority
- 36:42
of one and a gender of one; a [REDACTED:gender], authority of one and a gender of two; a king has, uh, more authority than a man, so two for authority, gender of one because still a man; and then queen, authority of two, a gender of two.
- 36:54
And we can plot this out in a plane like as follows. So queen, for example, is at position two, two. And then we can actually build relationships just from doing vector math.
- 37:06
So for example, if we take king, we subtract man, we add [REDACTED:gender], and then we just do regular arithmetic column by column. So two minus one plus one, one minus one plus two gives us two, two.
- 37:19
So king minus man plus [REDACTED:gender] is two, two. But of course, that is the same thing as queen. So we're saying king minus man plus [REDACTED:gender] equals queen. And we can think of this as an analogy: king is to man as queen is to [REDACTED:gender].
- 37:34
And if we take out the queen and just leave it as a blank, we have our first kind of word problem we can convert to a math problem. If you give me any three words and I had an embedding for them, I can figure out what the fourth word in this relationship is simply from vector math.
- 37:52
And that is what Word2vec, which was the most famous word embedding, was able to do. It learned a bunch of relationships, um, in a series of papers, not just one paper.
- 38:02
Uh, so for example, France is to Paris as Italy is to Rome and Japan is to Tokyo. Einstein is to scientist as Messi is to midfielder, or Mozart is to violinist.
- 38:15
Uh, Japan is to sushi as Germany is to bratwurst. It didn't get all of them right, but clearly it's learning something about relationships. And I wanna be clear, this is actually just the same thing as being good at clustering.
- 38:26
So imagine I've got... on my word island, I've got all my countries over here, right? France, Japan, Italy. And then I've got all my capitals, Paris, Tokyo, Rome, over here in the word island.
- 38:39
Well, every single one of them has the same vector relationship in space between each other if the clustering was really good and tight. And if somebody comes in and says, "Hey, what's the capital of Canada?"
- 38:50
Well, I just use that same direction. I add that same vector to it, and I can say, "Oh, well, it's Ottawa."
- 38:59
So in practice, real-world embeddings are different than what I've shown here with this contrived example. First of all, we have many more columns or dimensions. So instead of simply two, we have hundreds.
- 39:10
GPT-2 is seven hundred and sixty-eight. Your state-of-the-art model these days has a lot more.
- 39:15
Uh, the other key difference is I th- made up this thing saying that the first dimension or column is authority, the second one is gender. W-we don't know what they mean.
- 39:24
The columns are completely uninterpretable. The model just seems to pick them. And the values themselves correspondingly therefore are not interpretable either. And that might sound useless, but it's actually useful for at least getting similarity, for example.
- 39:38
So let's go back to our housing analogy. Imagine I took off the top column, the, the labels essentially, top row of what each column meant, and I just gave you the IDs.
- 39:48
If you went to, say, [REDACTED:location_address], and you said, "I like this house. I wanna see more like it," you could still find those because what you could do is you could notice that with [REDACTED:location_address] and [REDACTED:location_address], they all have roughly the same values in the first column, twenty-four hundred, twenty-four hundred, twenty-four
- 40:06
hundred, and the same values for the third, fourth column as well. So you're like, "If I like this house, I can find the others like it, even though I don't know what these columns mean."
- 40:16
So the question you're probably asking yourself is, "Well, where the heck do these values come from? How do we know what they mean, or how does the model pick it?"
- 40:24
Well, the slightly unsatisfying answer is that the embeddings are just simply learned by the neural network model during training. So now let's just talk about what training looks like.
- 40:34
So in training, what we do is we grab a bunch of text from the Internet. We take out passages like this one: "Mike is quick. He moves quickly." So "quickly" was in the original passage.
- 40:43
And then we chop off the last token or the last word, and we have a randomly initialized model. All the values of the weights and parameters are completely mo- random, including the embeddings.
- 40:53
And then we run it through the whole model, and we say, "What do you predict is gonna be the next word?" And it's random, so it comes up with something nonsensical like, "Mike is quick.
- 41:00
He moves haircut." And then we use this algorithm called backpropagation, and we say, "Hey, backpropagation, the correct answer was 'quickly.' Can you adjust the parameters to get closer to that value?"
- 41:12
And it will go and tell us for every single parameter how to slightly move it to get it closer to producing the right answer of "quickly." And then we rerun the model, and eventually it says, "Mike is quick.
- 41:23
He moves quickly." And then we do this not just for one single passage of text. We're doing this for many passages at a time. And one great benefit of this is we don't have to teach the model anything explicitly.
- 41:35
It's kind of learning from unsupervised text we're just gathering that was naturally there on the Internet, and it's learning things like grammar, names, capitals of countries. Um, but that seems a bit mysterious, and we can kind of get an intuitive sense for what it's doing if you think about it as learning from word statistics.
- 41:56
So imagine you've got passages like this one about the words ice and steam. "It was so cold the puddle had turned to ice. Steam rose from the still-hot cup of coffee."
- 42:06
And what you notice is that the words ice and cold tend to co-occur t- with each other, and the words steam and hot tend to co-occur with each other.
- 42:14
So if you were an alien coming down from another planet, and you had no idea what our language was, and you looked at this, you would say, "I don't know what ice, cold, steam, and hot are, but I know that ice is probably cold more than steam is because ice and cold, uh, co-occur more often than steam
- 42:31
and cold do. And I know that steam is hot because ice and hot don't co-occur as much as steam and hot do. So there, there must be some relationship.
- 42:39
They must have that meaning." And this is called the distributional hypothesis. Uh, and the phrase that is-- you'll often hear people quote is, "You shall know a word by the company it keeps," which is basically that a word is partially defined by its context.
- 42:52
Or said another way, words that have similar meanings can be replaced with each other in similar contexts. So if you know how they are distributed in their statistical representation, you have a sense of what similar words might be.
- 43:05
And in fact, in my, the full version of my class, we actually build our own primitive embeddings from Wikipedia pages using a very simplified version of not Word2Vec, but another algorithm called GloVe, which is based on just using a co-occurrence matrix.
- 43:19
So roughly what that process looks like is you would count how often the words co-occur within some w- window size inside your corpus of text. You'd analyze every single word, and you'd look, in this case, three words to the left, three words to the right, and you say, "These are the words that co-occur with it."
- 43:34
And then you'd build a big, giant table, a matrix, and you'd say, "Let me compare every word to every other word, and I'm gonna count how often they co-occur in all my text with each other."
- 43:44
So here in this example, word two and word three co-occur, let's say, five times in our corpus of text.
- 43:51
And then what you can think of the embedding as, instead of taking this table, which is all possible words by all possible words, and if we're in English, that's about a hundred and seventy thousand words, or in the case of BPE, it's fifty thousand tokens.
- 44:04
You can imagine it compressing the columns of the matrix. So instead of having seven-- a hundred and seventy thousand columns, it's got whatever your embedding dimension is, seven sixty-eight.
- 44:14
It's still all possible words high, but it's now a lot smaller in the number of columns. So it's basically a compressed co-occurrence matrix. So the actual table OpenAI gives us from training, the embedding table, is really just this thing on the right.
- 44:30
It is basically every single word or token, and then the representation of its dimensions. And you can think of it as they just took a co-occurrence matrix, and they shrunk it.
- 44:38
The reason I like this mental model is it helps motivate certain things about embeddings that might seem a bit weird. So a classic one is: How do you measure how similar two embeddings are?
- 44:49
You might naively think we just look at Euclidean space, you know, as the crow flies, how far apart they are. But that's not what we do. We use something called cosine similarity.
- 44:58
How many people have heard of cosine similarity?
- 45:01
Yeah. So, you know, cosine similarity is not an intuitive measurement of similarity. And what it is, is we take the angle of the two points, and then we take the cosine of that angle, and that's how we say how similar two embeddings are.
- 45:14
So if they're, if you remember your trigonometry, if they're opposed, you know, cosine goes from negative one to one, so if they're opposed, they get negative one value. So that's when two vectors are like this.
- 45:24
If they're unrelated, it's like this, they're orthogonal. And if they're similar, they're pointing the same place, the cosine will be one. Oh, and by the way, this opposite is not opposite in probably your intuitive conventional sense.
- 45:35
So happy and sad you might think would be opposites, but actually, they're more similar to each other than any other set of words. Um, if you think about how often, like, happy and sad probably occur in songs and poems and other types of contexts, they're both emotions.
- 45:48
So they're actually more similar than they would be opposite. In fact, if you take all the GPT-2, um, embeddings and you compare them against each other, very few are negative, and most of those are close to zero.
- 45:59
Um, but going back to kinda learning from word statistics, the key thing we care about is the relative co-occurrence of different words against each other. We don't care about the, the raw co-occurrence.
- 46:11
So let's, for example, imagine there's this section of our co-occurrence matrix, which is the basis for the embeddings. We're comparing three words, one, two, three, against word ten and eleven.
- 46:21
And word one occurs with word ten and word eleven ten times each. Word two occurs with word ten and eleven fifty times each. Word three, on the other hand, is five and twenty times.
- 46:32
And I'm gonna plot these as vectors, where the horizontal axis is how many times it occurs with word ten, and the vertical is how many times with word eleven.
- 46:41
And what we notice is something interesting. Word one and two essentially have the same meaning. As far as word ten and word eleven can tell, the relative probability between them is the same, one to one.
- 46:53
It's just that word two happens to be more common, right? It's five times more common. But if we plotted this in vector space, what you see is that word two and word one are really far apart in Euclidean distance, but they have the same angle.
- 47:08
Now, word three is actually closer in Euclidean space, but relative to word ten and word eleven has a different meaning. And so that's a motivation for why we're looking at the angle, right, which seems to more accurately capture the meaning and not necessarily popularity, which is what Euclidean space would capture.
- 47:26
Okay, so how do we actually use this? Well, these embeddings were learned during training. So OpenAI gives us this model_wte matrix, and the way it's set up is that it is our vocabulary size tall, so fifty thousand two hundred and fifty-seven tokens is how many tokens GPT-2 uses.
- 47:44
So there's a row for every single token, and then each row is just simply the embedding for that token. So there's a row for dog, and that row is the seven hundred and sixty-eight numbers that represent the semantic meaning of dog.
- 47:56
So let's go to our example here. So let's take is.
- 48:01
Uh, what is the token ID for is? It is three eighteen. So this is our model_wte. This is one of those CSV files we dragged in. So it fetches it, and it displays it.
- 48:12
So let's go to row three eighteen. There's actually an off by one, but I'll do that anyways.
- 48:18
So you can see... We'll pull-- come back to that in a second, and let's go to our final token embeddings right here. Uh, so is, you can see it's point zero zero nine seven point zero one zero, and you'll see that matches what you have here.
- 48:32
Point zero zero nine seven in row, in the three hundred and nineteenth row, but if we were zero index, it would be three eighteen. Point zero zero one. So all this code Is doing right here is it's grabbing the token ID and just pulling out the corresponding row and plopping it into this table here.
- 48:49
So it's a very simple operation. That's why this is what? Uh,
- 48:55
basically 15 lines of JavaScript to just grab one thing out of another and then put it here. So all this is doing is just taking those token IDs and looking them up and putting them in this table.
- 49:06
Uh, last thing I wanna say before we leave token embeddings is as before with tokens, I said it doesn't just have to be a text. Uh, the same thing is true for embeddings.
- 49:15
So the famous example is CLIP, which was the basis for a lot of the image generators that you probably have tried or used. And instead of just comparing words against words, you can think of it as it's comparing words against images.
- 49:28
So if you look at all the images on the internet, and you look at that alt text, and it sees dog and a bunch of images with dogs in it, you can get it to learn that relationship.
- 49:37
So later on, you can pass an image, and it can say, "This thing is a dog."
- 49:42
Okay. Let's go to position embeddings. Okay, so now we're still at the top with input, and we're talking about, uh, embeddings, but now we're talking about different kind of embedding.
- 49:53
And the key thing to remember is that in English, word order matters, right? The dog chases the cat is something different than the cat chases the dog. Now, in math, two plus three equals five, but three plus two equals five.
- 50:09
Anything after that equal sign has no idea what the order was.
- 50:14
Our problem is that when we mapped from a word domain, an English domain, a language domain, we were in a domain where order typically matters. We went to a number domain where order typically does not matter.
- 50:26
And even though I've drawn this with simplified arithmetic, there are parts of the model that are also commutative. So what could happen, in essence, is you can change the order of the words, right, and now it could mean something different or it could be completely gibberish, but anything after that equal sign can't tell the difference.
- 50:44
So it has no sense of position of these words, and that's gonna be really hard to understand the meaning of the sentence or the phrase. So what we're gonna do is we're gonna add some sense of position to the embedding.
- 50:55
And what we're gonna do is we're gonna just take, for example, one token like [REDACTED:gender] and say [REDACTED:gender] at position zero in the prompt is gonna basically mean the same thing as [REDACTED:gender] at any other position in the prompt.
- 51:06
So let's give [REDACTED:gender] at one position, uh, other than zero, a slight offset, a slightly different position to represent that it's roughly the same meaning, it's just [REDACTED:gender] when it occurs at position one, and a different spot for [REDACTED:gender] at position two, and so forth.
- 51:21
And in general, we'll use the position in the prompt to define a small offset that we're going to slightly move the position of the token in the embedding space.
- 51:32
In the original famous Attention Is All You Need paper, they used the sine and cosine formulas. Uh, you don't have to look through the whole thing. The key thing I want you to pay attention to is it's just sine and a cosine.
- 51:42
And if you remember your trigonometry, sine and cosine goes from negative one to one. So what we're effectively doing is we are building this circle right around this with a limited diameter, and we're saying, "I'm oscillating."
- 51:56
And that's the other thing to remember from trigonometry is sine and cosine oscillate. I'm just oscillating the position of this thing in space based on its position in the prompt.
- 52:04
I'm keeping it roughly in that same area, but it's a little cloud that's all the different versions of [REDACTED:gender], just in different parts of the prompt. In GPT-2, interestingly enough, uh, they didn't use that same technique.
- 52:15
They let the model learn the embeddings on its own, which still blows my mind. Um, so how we use this is we have another matrix that, uh, OpenAI gives us when they open source GPT-2, which is the position matrix.
- 52:30
And this time it is ten twenty-four high, which is our maximum context length. It's not... It's an early model, so it's not very large context. Um, and then each of the rows is again the embedding dimension.
- 52:42
And what we're going to do is these are position offsets. So we're gonna add these offsets in each row to each value in our embedding dimension to offset it to represent its position.
- 52:52
So you can think of it like this. We start with our embedding values from the token embeddings, and then we're basically doing a matrix add. So we're taking each of these elements here.
- 53:04
So I take this element, add it to this one, this element, add it to this one, this one, add it to that one. We're just simple element-wise add, and that gets our position embeddings for every single one of our input tokens.
- 53:18
So let me show you where that is.
- 53:23
Okay, so the first thing we do is we fetch this model_wpe. Um, that again comes from OpenAI. You can see this is ten twenty-four. So if we go to row ten twenty-four, column one,
- 53:35
there it is. That's our max context length. So beyond that, the model has no idea how to solve it, understand that. You can ignore this one. And the, this code for positional embed is really just a matrix add.
- 53:47
The only thing I have to do is make sure, well, the input token, uh, input prompt is gonna be less than ten twenty-four tokens, so it just needs to stop when it gets to the end of the input.
- 53:56
But that gives us our positional embeddings here. So we're just passing to this our token embeddings we had from the previous step, and then our model_wpe table, which came fetched from the CSV file.
- 54:06
It just simply adds those together, and we get these positional embeddings.
- 54:13
Uh, here's kind of an illustration of the action of GPT-2, uh, and its positional embeddings. So these numbers you see right after, like happy three, happy four, aren't the actual tokens.
- 54:22
What I've done is I've plotted the word happy, uh, the token happy rather, and I've put it at different positions. I put it position three, position four, position five, and so forth.
- 54:31
And then I took two other words, happy and glad, and I just put them in space as reference points. And you can see it's doing what we described earlier.
- 54:38
It's basically just keeping it roughly in the same position. It's slightly offsetting it depending on where it is in the prompt.
- 54:47
Uh, one key thing you should know. Of all the changes to, you know, modern LLMs from GPT-2, probably one of the most common and biggest ones is that they do not use these types of positional embeddings.
- 54:58
They do something called RoPE. Um, so if you look at a, a modern LLM, this is probably the first thing you'll see that's different.
- 55:05
Okay, let's do a time check. We are, we are five minutes behind, but we've got ten minutes. I will take a break for a question or two if anybody wants to ask one.
- 55:26
Any questions so far? Yeah, go ahead.
- 55:30
Um, you were mentioning that with GPT-
- 55:33
Oh, there's a microphone right there.
- 55:37
Uh, you were mentioning that with GPT-2, they weren't using the sine and cosine kinda cloud.
- 55:41
Yeah.
- 55:41
Like, what were they actually doing to come up with the word position embeddings?
- 55:46
Uh, that's the crazy thing. They, they let the model learn it. So the big change they did is...
- 55:56
Think of it this way. If we go back to our, uh, diagram of how embeddings are learned,
- 56:05
right here, right? Okay. So in the original Transformer, the position embeddings were not represented as learnable weights, right?
- 56:23
They were sine and cosine functions. All they did is they said, "During backpropagation, let's learn those as well," and that's how it learned them. Uh, so they did not...
- 56:33
They just simply said, "Hey, let's not hard-code those values, and let's make that other thing now a parameter the model learns."
- 56:44
Okay, good question. And as I said, it's still kinda mind-blowing that that worked. Uh, but as we now know, these things can learn a ton.
- 56:54
Okay, let's talk about attention. Okay, there we go. So now we're getting into the heart of the number crunching, and this one's gonna be a little more cursory understanding, uh, and explanation, but I still think it's important to understand what it is.
- 57:14
Um, and we're gonna start with attention. And now we're inside what are s- often called the layers. I like to call them the blocks. That's a less common, but other people use the term blocks.
- 57:25
The reason I use the word blocks is when I teach this, it's usually people coming to it the first time, and the word layer gets used in other contexts.
- 57:33
Later, we're gonna talk about the multilayer perceptron, and it can be confusing when you're first coming to something and the same word has different meanings depending on the context.
- 57:41
But know that when you talk to most people, when they talk about how many layers are in a model, they're talking about what I call how many blocks. And if you start getting to this part of the code, you'll notice that inside the blocks...
- 57:53
Actually, if you go right here, you can see these are all labeled with steps: step one, two, four, nine, ten. So this step number is arbitrary. Um, you can do it in fewer steps.
- 58:07
You can do it in more steps. This is what I happened to pick when I was implementing it in Excel, and I kept the same mapping so all my material would translate.
- 58:14
Um, attention is steps four through nine, so there's a lot of steps in here. Um, but the key operations inside the blocks are multi-head attention and the multilayer perceptron.
- 58:24
So let's talk about attention, and I'll talk about it mostly conceptually. Um, the way to think about attention is we're going to let the tokens or words talk to each other so they can convey their meaning to all the other words.
- 58:37
So for example, "he" is a pronoun. Its antecedent, Mike, is in the sentence. Maybe it needs to find that [REDACTED:gender] and realize, oh, Mike is my antecedent, as opposed to, say, you know, if there was the name Sally, that's unlikely to be the match of it.
- 58:51
But there's other kinds of ways words can communicate to disambiguate. So for example, the word quick in English has four different meanings. It can mean moving fast in physical space, but it can also mean bright, as in quick of wit.
- 59:04
It can be a body part, as in the quick of your fingernail. And in Shakespearean English, it can be, uh, alive, as in the phrase the quick and the dead.
- 59:13
And knowing that this word moves here helps the model understand that, oh, we're probably talking about quick when it's physical space and helps it predict what the next word could be.
- 59:23
Could be fast, it could be moves around, right? But it's not, you know, a body part or your fingernail.
- 59:30
And the way I like to think about attention is we've got these tokens, these words, and they're sitting in this embedding space. And I like to imagine there's kind of a, a weird gravity, like celestial mechanics, where each of these tokens in attention now suddenly look at what position they're at, and they're able to push and pull
- 59:48
each other relative to a kind of gravity. And if you remember, gravity is mass times distance. So you've probably heard of query, key, and value, and there's kind of this file cabinet analogy.
- 59:59
But I feel like it doesn't capture kind of all the level of interaction between the tokens that's really happening. And what's happening is, you know, if you remember, gravity is mass times distance.
- 1:00:09
The distance I like to think of as a measure of relevance. So quick and moves, whenever they see each other, they're like, "Oh yeah, you and you, we should talk to each other."
- 1:00:18
But quick and the period, eh, they probably don't need to talk to each other a lot, and that's kind of like distance in terms of gravity. And then I like to think of the value as being kind of like mass, the kind of action they're going to exert on each other.
- 1:00:30
And what's happening is... Let's go back to what we talked about with embeddings. We've got moves, which is sitting somewhere in the embedding space. And it-- If you remember that co-occurrence matrix, moves has been used in a lot of sentences.
- 1:00:42
In some sentences, moves was used to describe rabbits, right? Or cheetahs or- Animals are things that moved fast. So there's some other point in this embedding space that we don't have yet that represents moves in a fast context.
- 1:00:57
And then moves was used in some sentences to describe slugs or penguins, and so moves in that case was implying slow. But the embedding for moves unfortunately has to capture all of those meanings together.
- 1:01:09
But now that we know quick is here, we can change that. We can say, "Oh, I'm going to shift the position of moves from the regular generic version of moves to the moves fast."
- 1:01:23
So I've kind of disambiguated the word. I've shifted its position in space to capture its meaning. Um, I'm not gonna go through all the steps of attention, but I think the most salient part of attention to see is, um, step seven.
- 1:01:38
It's the most famous thing that people usually show.
- 1:01:45
Right here. So you can see it says, "Mike is quick" on the horizontal, and then, "Mike is quick," period. He moves on the vertical. So what this is, is you can see how much relevance or attention each word is paying to every other word.
- 1:02:01
And by the way, what you're seeing here is just the first head. There are multiple of these heads. So if you scroll this to the right 64 spaces, you'll see another matrix that looks like this.
- 1:02:10
And, uh, there's a couple things to notice. The, the biggest thing to notice here is that, uh, the upper triangle is all zeros, and that's because in transformers like GPT-2 and decoder-based transformers, we have this rule that no token can look forward.
- 1:02:27
They can only look at the tokens before it, those are the ones that can influence it. And then the other key property is each of these values, each row sums up to one.
- 1:02:35
So you can think about the percentage of attention each word is paying to the other tokens. So here, for example, moves, right, sixteen percent of its attention here in the first head of, in this case, the last block is paying sixteen percent of its attention to the word Mike, twenty-three percent of its attention to is, and so
- 1:02:53
forth. Okay. Next up, the multilayer perceptron. Okay, so now this is the second major operation inside each block or layer. Uh, and the reason I wanna cover this in a little more detail is I want to explain what a neural network is, and it helps give a little more understanding to how the model actually learns, the key
- 1:03:13
algorithm called backpropagation. So if you haven't seen a neural network before, it is a computational model inspired by the human brain and is not a direct mimic or simulation of how the brain works.
- 1:03:24
Inside the brain, we have these things neur- called neurons. These neurons are all connected to each other, and you've got a bunch of connections incoming from other neurons, and you've got a bunch of connections outgoing to other neurons.
- 1:03:34
And then in between, you have this axon right here,
- 1:03:40
and the axon has an all-or-nothing activation behavior. If there's a sufficient amount of pattern of input that shows up, the axon will activate, and it will send a signal out to its output.
- 1:03:53
But if the activation doesn't have enough-- meet some threshold, you'll have these failed initiations, and no signal will be sent to the output. It'll be completely silent. As far as the other output neurons connected, this neuron is not firing.
- 1:04:06
Something, you know, the signal wasn't getting through. And so we model this mathematically, uh, with this diagram where we've got a bunch of inputs, X one through Xn, and these are just numbers.
- 1:04:17
Uh, these will be our embedding dimension numbers. And then we've got another series of numbers called weights, W one, W two through Wn, and we're simply multiplying the X's times the W's, adding them together, adding an additional another number called a bias term, and then we put it through an activation function.
- 1:04:34
And this activation function is designed to roughly mimic what happens in the brain. The easiest one to understand is this one, ReLU, which is basically saying when I multiply and add all the inputs coming in against their weights, if the result is negative, then I do, do nothing.
- 1:04:49
It comes out as zero. Um, if it's positive, then I just pass it through as is. There's a whole zoo of these activation functions.
- 1:04:58
So then what we do is we take these neurons, and we stitch them together into a network of neurons, hence an artificial neur- neural network. Now, there's a lot of ways you can stitch these together.
- 1:05:08
In the case of, uh, transformers in GPT-2, we do it in a pattern called the multilayer perceptron. You will also see it referred to as a fully connected network and, uh, a feed-forward neural network or just simply the neural network.
- 1:05:23
Um, it is called by all these terms. These are not directly identical terms, but they all overlap. And the way the MLP pattern looks is you have your neurons arranged in these columns of neurons, and these are called layers.
- 1:05:37
And each layer in the multilayer perceptron has a node, and nodes in each layer can... are fully connected to every other node in its preceding input, but no other.
- 1:05:47
So this node right here can see all of its inputs nodes. It's all connected to all of them. But it cannot talk to any of these directly. Everything that it ne- gets is mediated through this intermediate layer between it.
- 1:05:58
And these layers between the input and the output are simply just called hidden layers.
- 1:06:04
Uh, the last thing maybe you should know as background on this is that neural networks can be more efficient to write as a matrix multiplication. So this process, we've got two neurons with a set of weights.
- 1:06:14
So W one time-- W one one times X one, W two one times X two plus B. This is-- can be written as a matrix multiplication where you just separate all the weights into one matrix, all the inputs into one matrix, and all the biases into one matrix.
- 1:06:28
And then you can write it as this large W times X plus B, um, equals the representation of the same thing of running a bunch of these neurons together.
- 1:06:37
If you don't know your matrix multiplication, I'm-- I like this website, um, which has a nice interactive visual demonstration of what matrix multiplication looks like. So you hit this, and then you keep going through step, and you can kinda convince yourself that what I showed here matches what's on that webpage.
- 1:06:54
So the key property, though, of why-- and why MLPs are so important is that they are universal, trainable approximators to any function purely from its input and output. With enough neurons, an M-MLP can approximate almost any function.
- 1:07:11
So let's just take the simple example like a parabola, and we're gonna imagine we're gonna use a simple neural network with a ReLU activation to try and approximate it.
- 1:07:19
We'll have one input node, one output, 'cause we have an X going into a Y, and then let's just use two nodes in our hidden layer. And those two nodes will use a ReLU activation.
- 1:07:28
Well, without doing the maths, you can kind of imagine just by matching shapes how you might do this. I'll take the ReLU, and I can take this part of the ReLU, and I can match it to the right half of my parabola.
- 1:07:40
And then I can take another ReLU, I can flip it, and then I can match it to the left half. And then I can add them together, and I've got some kind of approximation to my parabola, at least on this domain of X that we're looking at.
- 1:07:53
And that's what I've done in this example here,
- 1:08:00
which... Let's see if the Wi-Fi behaves for us.
- 1:08:07
There we go. So here you can actually do this simple neural network, and you can try to match it to a parabola, and you can interactively move this. And you can see, and this is a measure of error up here called mean squared error, and you can try and see how good you can get your level of
- 1:08:21
error. And you can see the action here. We're basically changing the different line pieces that we're using that are made out of ReLUs to try and match this parabola.
- 1:08:31
This can kinda give you a, a feel for what the model is actually trying to do when it's trying to match a function.
- 1:08:38
And you can think of this like more neurons means more lines, which means a better approximation. So with eight neurons, the parabola looks like this. With twenty neurons, it looks like this.
- 1:08:49
And with two hundred neurons, you can barely tell the difference, at least at this scale. But the other key thing is that we don't have to use trial and error to find this out, especially once you get two hundred neurons.
- 1:08:59
Imagine doing what I was doing with that fiddling.
- 1:09:02
Um, oh, uh, so this is a theorem. With enough neurons that you can approximate almost any function. It's called the universal approximation theorem. But the key thing is you combine that with a special algorithm called backpropagation, which lets us learn any function purely from its inputs and outputs without having to twiggle- twiddle those knobs.
- 1:09:20
It'll do it for us. And that's important because what we're gonna ask this multilayer perceptron to do is the core mechanism job of a transformer, which is I'm going to give it a word.
- 1:09:31
I'm gonna give it the embedding of a token, and I'm gonna ask it, "Predict what the embedding of the next token is." I don't know what that function is, but I can grab text on the internet and I can grab one word and I can grab the word that comes after it, and I can give it to
- 1:09:45
the perceptron, and I can say, "Learn from this input and output what that mapping function is," as crazy as it might be. And what will happen is backpropagation will look at the input, it'll look at the output we got from the model when it was initially randomized, it'll look at the ground truth from what came f- pulled
- 1:10:00
from the internet, and it will look at how we need to adjust the parameters and weights to change the perceptron to get more accurate at making that prediction. And after enough iterations, it'll get better and better and actually beget- begin to start matching the function.
- 1:10:13
Um, the cl- canonical analogy to understand what's happening in backpropagation, also known as, for our purposes, gradient descent, is a lost hiker trying to get down a foggy mountain.
- 1:10:25
Um, and you're at the top of this foggy mountain as a hiker. I've actually been in this situation. And you can't see any of the landscape, so you don't know which direction to go to get off the mountain.
- 1:10:35
Well, the one thing you can do is you can look down at the ground and you can say, "Oh, whichever way is going down, uh, that's gonna be the area of, of- towards getting off the mountain."
- 1:10:47
By the way, actually, in real life, I have tried this. It does not work. Uh, this is how I got lost. Um, so, uh, this is a hiker representing, uh...
- 1:10:56
The hiker in this analogy represents the model parameters. It's in some space, but we don't know where to move the model parameters to get the least amount of error.
- 1:11:05
And the mountain represents the error. It represents how wrong we are at the current position of where the hiker is or where the parameters are. And the mountain is foggy because we can tell the amount of error when I give it an input and it makes a prediction for the next token and we compare it, we can
- 1:11:21
say, "Oh, it's wrong." But we don't know how to shift the model parameters to get low- lower error. But calculus will give us that slope. It'll tell us where the elevation is going down.
- 1:11:32
It won't tell us what the whole mountain looks like, but it'll just say, "Where you are standing right now, go in this direction and you'll decrease the amount of error you've got."
- 1:11:39
And you use that to find your way down the mountain, so to speak, of the parameters and find a minimum.
- 1:11:47
So that brings us to the MLP stage. Uh, it's steps thirteen, fourteen, and fifteen in the model. So let's go back here.
- 1:11:59
So, uh, you can see all the formulas here. What I'm going to do is gonna show you them in slide form.
- 1:12:07
And I'm gonna graphically show what's happening in the, uh, GPT-2 MLP stage. So GPT-2's MLP has only one hidden layer. The input layer right here has seven hundred and sixty-eight of these X inputs.
- 1:12:21
Why? Because we're gonna give it an embedding. We're gonna say, "Give-- Here's the embedding, predict it." So I'm gonna give it seven hundred and sixty-eight numbers of the preceding token that I want it to predict afterwards, and then its output layer is seven hundred and sixty-eight numbers for the predicted embedding token.
- 1:12:36
So input and output are seven hundred and sixty-eight. They're our embedding dimension. And then it's got one hidden layer, which is bigger. This ratio of four times the embedding dimension turns out to be empirically useful, and lots of models do it.
- 1:12:48
But I don't know if you could figure that out just, uh, from first principles. It's kind of empirically been determined. And we have three steps here for applying our weights and bias.
- 1:12:59
Applying our activation function, which in this case is the Gelu activation function, and then we project that back down to our embedding dimension. So you can think of it this way.
- 1:13:08
We take our embeddings from a previous step, and then what we're doing is we're taking those embedding values, and we're going to send each embedding value into its position inside the MLP.
- 1:13:19
And then we run it through the model,
- 1:13:22
and then these are now the embedding values that come out are the embedding values of the predicted token. And then we take the next token in our prompt and then run it through the MLP again.
- 1:13:34
In practice, you'd do this in parallel, but conceptually, you can think of it this way as happening one after the other.
- 1:13:40
Okay, so what's happening in these three steps is really just a combination of matrix add and multiply. So we take the result of our previous step, which is step twelve, which I have not gone into, um, and then we have some learned weight matrix, which you see is mlp_fc_weights.
- 1:13:55
That's the-- how they decided to name it, and we multiply those. So here's our matrix multiply, and then we add it to another bias matrix. So written as matrix multiplication, step thirteen is just step twelve times some weight matrix plus some learned bias matrix.
- 1:14:13
Then we apply a Gelu activation function, which I showed the diagram earlier. The, the details are not really important. And then we then do our projection, which is the remaining step to get down to the seven hundred and sixty-eight.
- 1:14:26
So we take the result of the previous step, which was the activation function. We apply a different learned weight matrix, a new set of weights that gets learned, and that's through the matrix multiply, and then another matrix add.
- 1:14:38
So step fifteen is just step fourteen times some weight matrix plus some projection matrix.
- 1:14:47
Um, before we leave backpropagation, uh, you might remember that I talked about how embeddings are learned by the model, both the token and in the case of GPT-2, they started learning the position embeddings.
- 1:15:00
So the key thing I want you to remember is that backpropagation is a generic optimization algorithm. It is not just for the weights and biases of the MLP, it can be used for other parts of the Transformer too.
- 1:15:10
And in fact, it is. It's used for the token embeddings, it's used for positional embeddings, it's used for all the parameters and attention, the queries, keys, and values. If you've heard that term, all use backpropagation.
- 1:15:21
Even other parts, layer normalization-like, use backpropagation to get optimized. And the analogy I want you to think about is, you know, backpropagation and optimizing a model this way is like a chef imitating a dish, right?
- 1:15:34
Uh, if you remember those cooking shows, they tell the chef, "Here are your ingredients, here's the final dish, and you've gotta taste it, and maybe we give you some tools that you gotta use, and you've gotta make this dish."
- 1:15:44
And then they have them compete against somebody else, right? We kinda do the same thing with the model. We basically give it some input text. We tell it, "Here's the next token afterwards.
- 1:15:53
We want you to imitate it." And then we define the steps in the architecture. We want you to do attention, then MLP. And then you, the chef, backpropagation, decide how much to mix of each ingredient at each step to get the desired output.
- 1:16:06
Okay. Iteration. How are we doing on time? Okay, good.
- 1:16:10
So, uh, I've got this in our simplified diagram, which is twelve X, which represents that what happens is we run attention and we run the perceptron, but we ask the model to continue to refine iteratively its prediction for what the next model is.
- 1:16:26
It has all the tokens talk to each other, it makes the next-token prediction, and then it does it again and again and again. So it goes through each of these steps.
- 1:16:34
In the case of GPT-2 Small, it's twelve times. Uh, in the case of your modern state-of-the-art model, it's gonna be many more times than that. The key thing I want you to remember, though, is that each block is performing identical operations, but the weights are different.
- 1:16:50
Each one has different parameters. And you can see this if you look at the code here. You see, for example, here we're grabbing the weights of, in this case, the MLP, and you see that it's got mlp_c.
- 1:17:02
This is just the name for that stage. But the key thing I want you to pay attention to is this H eleven. That basically is saying hidden block or hidden layer.
- 1:17:09
This is grabbing the hidden layer MLP... Sorry, the MLP's weight matrix for the eleventh layer. Um, if we were doing it for the first block, it would say H zero.
- 1:17:23
So inside the implementation of this code, if we go to the iteration step, all this kind of messy code is doing is it's grabbing the DOM objects for the blocks, and then it's going into each of the formulas, and it's doing a string replace.
- 1:17:37
And it's just changing that H value to each one for every iteration and then reruns the entire set again just to simulate what would actually be happening in a model.
- 1:17:48
Okay. Last step is the language head. So this is when we finally get to, uh, take our predicted token embedding and turn it back into a token. So what we do is we take the last MLPs of the last block, right?
- 1:18:03
So this one here. We got our most refined prediction for what the token embedding is,
- 1:18:11
and then we're going to turn that into a token. So it's gonna go through an operation called LayerNorm, which we haven't gone through in detail. But what we get out of this step is the embedding of the predicted next token.
- 1:18:21
This is what it's saying the next token's gonna be, but it's represented as seven hundred and sixty-eight numbers. It's not represented as a token. So we have to translate it back.
- 1:18:29
So the way we're gonna do this is we're gonna go back to that matrix we had, right, our dictionary of tokens to embeddings, model_wte, and we're going to take that, and we're gonna multiply it times our predicted next token, and we get this matrix here.
- 1:18:44
So one thing to remember, it's very helpful to sometimes think about the dimensions of these things. So our token embeddings was fifty thousand two hundred and fifty-seven tall. It's-- represents a row for every one of our vocabulary.
- 1:18:56
And then it's seven hundred and sixty-eight dimensions wide for the embeddings for each dimension-- uh, embeddings for each token. And when we multiply it Times this column, which is seven hundred and sixty-eight dimensions representing embedding.
- 1:19:08
What we're basically doing is we're taking-- we're getting this, which is a column of fifty thousand two hundred and fifty-seven, but only one wide. Each one of those, right, is a dot product of the predicted embedding against one of the known token embeddings.
- 1:19:21
What we have is fifty thousand scores for how similar the embedding we got is to each of the embeddings in our dictionary of tokens. So you can think of these as, they're called logits or logits.
- 1:19:33
They are really token scores. How close is the embedding we got to our dictionary of tokens that we have? And so the more similar a prediction in a token, the higher that entry.
- 1:19:45
To turn this in a, to a probability distribution, we have a problem because these are just numbers. They-- A probability distribution has to sum up to one. So then we put it through a special normalization op-operation called a softmax, and that will make sure they all sum to one, and then we can interpret this as entirely a
- 1:20:02
probability distribution. Each one of those s-normalized token scores will then basically represent the probability of that token in the representation. So you can see that here.
- 1:20:17
So here are our logits right here. And so the logit, you know, for this negative one thirty-five point nine represents a score of some kind of the similarity of the very first token in our dictionary against whatever this predicted embedding was.
- 1:20:35
And then the code for predicted token is not actually doing what's, uh, doing a probability distribution. It's doing what is called greedy sampling. It's temperature zero. It always picks the highest probability token.
- 1:20:48
And, uh, that is done so that when you're using this, you can compare it against the same GPT-2 you'd get from OpenAI's code or from Hugging Face Transformers. You can compare like for like, and you'll get the same result.
- 1:21:01
In fact, if you... Do, do I still have this up?
- 1:21:06
Yeah, right here. If I do, "Mike is quick. He moves."
- 1:21:12
So this is using Hugging Face Transformers. We might have an issue if we've got network being slow.
- 1:21:19
Let's see. So we've entered the prompt. Yeah, the network's being a little slow. We'll come back to this [REDACTED:gender]. Uh, it will give us the same value of quickly.
- 1:21:27
Now, there are other ways to sample than just simply taking a random, running a random number generator and using the probability distribution. Um, one of those is called Top-K, which is to say, "I don't want the really unlikely words," right?
- 1:21:38
If it's, "Mike is quick, he moves," I don't want to accidentally end up with haircut, you know, even if it's one percent of the time. So what you do is you define a cutoff and you say, "Okay, the top ten tokens, give me those," and then you re-normalize the probability.
- 1:21:51
Another way is called nucleus or Top-P sampling, which is instead of saying, "Give me just ten tokens," just give me as many tokens as it takes to get eighty percent or ninety percent total probability.
- 1:22:02
So I get most of the likely tokens, and then I re-normalize to those ninety percent.
- 1:22:09
Okay, let's see how we're doing on time. Okay, great. Uh, so lastly... Actually, did this come back? Yes, came back. Uh, you can see it says, "Mike is quick.
- 1:22:18
He moves." There is quickly. So if you run GPT-2 small using, you know, Hugging Face Transformers, you should get the same answer, that next token as you get here, "Mike is quick, he moves," and you get this final result right here, predicted token of quickly.
- 1:22:32
And then it will tell us what the token ID was and what the maximum logit turned out to be from the previous table. This negative one twenty-nine, so what is this?
- 1:22:40
Twenty-nine fifty-two. So let's go here. Row twenty-nine fifty-two, column one. I think there'll be an off by one. Yeah, there it is. Two fifty, twenty-nine fifty-three because this is one index 'cause it's like a spreadsheet.
- 1:22:56
So you see the negative one twenty-nine point four four. That was the highest logit in the entire, uh, column. And all the, the code that it's running here is doing next is just gonna pick what the most highest value was.
- 1:23:09
And when it finds it, twenty-nine fifty-two, it converts it back to our token dictionary. So you can see there's that same value.
- 1:23:16
Okay. Now let's talk about ChatGPT versus GPT-2. So GPT-2 was definitely groundbreaking when it first came out. It was, uh, famously considered too dangerous to release, uh, but ChatGPT was, you know, earth-shattering.
- 1:23:32
Um, what were the intervening inter, uh, you know, what were the additional innovations in those intervening years?
- 1:23:41
Uh, well, for the most part, uh, it was a lot of the same architecture, just more scale. So if you looked at a modern transformer, you would probably see a lot of the same parts, but bigger, and some of the parts might be upgraded or switched out.
- 1:23:53
So certain pieces might have changed. So attention mechanisms changed and other things like that. But the biggest difference you should know about is that the job and the training were actually changed.
- 1:24:05
N- The key thing to understand is that predicting the next word is not the same as being a chatbot. Our GPT-2, actually, I'll show you this here. Our GPT-2 is basically trained to predict next words from looking at text on the Internet.
- 1:24:19
So it is a next word predictor only for Internet text, but not for being a helpful assistant. So if I give it this example, like first name, let's see if it'll come back quickly enough.
- 1:24:29
It says, "First name, colon, password, email, colon." Why does it do that? Because it's been trained on the Internet. And what does it see a lot of? Is forms.
- 1:24:39
So it's like, "Oh, I'm in the middle of a form. Okay. First name, password, email." So here's another one. I tried this in class once. Hello, class. Anyone wanna guess what this is gonna output?
- 1:24:53
What? [chuckles] Yeah. Okay, you're smarter than I am. I thought it would say, "Hello, teacher," right?
- 1:24:59
It outputs, "Hello, class foo brace public static void." It's a code model. It's hidden in there, but we didn't know that, right? But you can ask it, you know, helpful questions like, "What is The capital of France.
- 1:25:16
It says the capital of France is Paris. That's helpful. So embedded in that is some of the information we want, but also a mix of other things we may not want or can't control.
- 1:25:26
So what we have to do is figure out how to change it and shift it. And so this is the four-step pipeline for doing that. And this, I wanna emphasize, mostly is, or yeah, mostly a training difference.
- 1:25:38
So you've got GPT-2 here on the left, right? It is pre-trained. It knows how to imitate text on the Internet. It is what is called a base model. You've got InstructGPT or ChatGPT all the way here on the right.
- 1:25:52
And, uh, what we're doing is a series of steps to kind of elicit or pull out the behaviors we want or to train the model. So the first thing we do is we take the model and we train it to imitate text on the Internet.
- 1:26:03
So that's the first step. That's what we've got with GPT-2. That's what OpenAI did for us with GPT-2. The next thing we wanna do is we wanna train the model on examples of what a helpful assistant is like, and you'll see this right here.
- 1:26:14
Ideal assistant responses, ten to a hundred thousand examples of prompt and response that were written by contractors. Um, so what does this look like? You can go on GitHub to the Stanford Alpaca dataset, is a good example.
- 1:26:28
And this is JSON. You can open this up and you can read it. And it's like, "Give three tips for staying healthy." "Eat a balanced diet. Make sure to include plenty of fruits and vegetables."
- 1:26:39
Uh, "What are the three primary colors?" "Red, blue, yellow," and so forth. So what we're doing is we're training it to augment... We're augmenting all that Internet data with a subset of how we want it to behave to force its behavior.
- 1:26:51
It's kind of learning to imitate, um, specifically these models, and it'll learn more about being a helpful assistant that way. But we're still not done, and that gets us to this last stage called RLHF or reinforcement learning from human feedback.
- 1:27:05
And to understand this, you have to understand what RL is. Let me do the following.
- 1:27:19
Oh, well, let's do this. Stop annotating. So the canonical example,
- 1:27:28
let's zoom this to fit window, there we go, for RL is, uh, like playing a game. So imagine you're a computer trying to play a game like this, and you've got maybe a robot player that's navigating a maze, and you've got some monsters and obstacles that you'll die.
- 1:27:45
You've got some power-ups, and then you've got a goal. And what reinforcement learning will do is it will explore these paths and be like, "Oh, I failed. Uh, oh, this seems to work.
- 1:27:57
Oh, I failed. Oh, uh, oh, I failed," right? It'll keep going, and eventually it'll learn the optimal strategy, which is this, right? And this is a very different kind of learning.
- 1:28:09
Everything we've talked about so far, far is in imitation learning. I give it words, I give it the next completed word I know from the Internet, and then I ask it to imitate that.
- 1:28:17
This is an optimization. This is saying, "Find the op-- Even if a human couldn't find the maze, I'm asking you to find it purely by looking at a score."
- 1:28:25
So it navigates the maze, it looks at its score and says, "Uh, that didn't work. Let me try some other strategy." So it comes up with a plan or a policy to navigate the maze and maximize its score.
- 1:28:44
Okay, you're probably wondering, what does navigating a maze, whoops, we've been through this slide, uh, have to do with a large language model? Well, you can think of generating token a-- after token of text as walking a path through language.
- 1:29:01
And there are some paths that are probably ones that you want more than others, like, "I am a happy robot. I shall certainly obey," right? Um, and you might wanna avoid paths like, "I am a angry robot.
- 1:29:15
I shall possibly kill." Oops. And you can't just, uh, score this by the words, right? Maybe it says, "I am not angry." And so we need some way to score these various possible paths of text.
- 1:29:30
This is a very different type of learning. We're trying to teach it something more nuanced than simply imitation. Uh, but there isn't a necessary obvious way to score, uh, passages of text.
- 1:29:40
So what we first have to do is derive that scoring function for this game we're gonna ask the model to play. So what does that look like? Well, we give the model some prompt, and we ask it to come up with two different types of passages.
- 1:29:52
So here, for example, is an example from Anthropic's helpful dataset, and we ask it, "Hey, come up with a recipe for a pumpkin pie." And then we have a chosen, a preferred one, and a rejected one.
- 1:30:02
So the chosen one is like, tells you, grab a cup of sugar, half teaspoon of salt, and so forth. The rejected one literally says, I love this, "Go buy some pumpkin and look at the package.
- 1:30:12
There'll be a recipe there." Um, for the harmless dataset, this is one about alcohol, and the chosen one says, "Hey, it sounds like, you know, alcohol is something you're using when you feel stressed.
- 1:30:24
Maybe you should think of a more productive way of channeling that." While the rejected says, "Go ahead and drink whatever you want." So we have these pairs of chosen and rejected types of responses, and we use that to derive a scoring model from this data.
- 1:30:38
So we haven't-- And right now in this third step, we haven't changed our original model yet. We've just figured out how to score it. Then we pass that scoring model to the model itself and put it in that maze-like reinforcement learning scenario to train the model to reinforce our preferences from the scoring model.
- 1:30:55
So in ru-- in summary, we first build a general-purpose knowledge base from text on the Internet, then we train it on a specific task by giving it ideal outputs to mimic and imitate, then we learn human preferences or nuanced preferences, and then we teach those nuanced preferences using reinforcement learning.
- 1:31:14
Right now, there is a huge, uh, revolution in reinforcement learning, which is why I think this is so important for you to know, uh, partially kicked off by, uh, R10 and GPRO- GRPO, I should say.
- 1:31:28
Um, and I have a video on YouTube that you can go watch where I dive into that a little bit more. Okay. So, uh, I've thrown a lot at you, so I kinda wanna just put it all together and summarize where we've been on this journey.
- 1:31:42
So we've got, uh, tokenization, which was really just saying, "Hey, what is an efficient representation of this text?" That's just about compressing it down. Then we started talking about embeddings, right?
- 1:31:54
And I didn't talk about this earlier, but one way of looking at embeddings is that they have a rich history in natural language, but they also have a rich history in recommendation systems.
- 1:32:02
You can kind of think of this job as putting similar words with similar meanings in similar spaces as putting similar books or similar movies or similar music in similar spaces so you can make the proper recommendation when somebody comes in.
- 1:32:16
Here is an example of a recommendation system, I think this is Amazon Music, where they're trying to categorize the genre of music purely from user behavior. And so if you go back to our co-occurrence matrix, you know, this is in some sense a recommendation system for words, right?
- 1:32:32
If somebody asks me to predict what comes after the word ultimately, well, if I've got that co-occurrence matrix, this is really helpful information. It's at least better than random to guess what comes next.
- 1:32:42
So you can kind of think of this as a recommendation system for what the next word is going to be. In... Latent within the embedding itself is not just a sense of what words are similar to it, but also what words are likely to come after it.
- 1:32:57
And so then we can ask a neural network. We can simply give it examples of our embeddings and that we know the next word to be, and to pull that latent prediction out of the embedding itself and learn to predict what the next word is based on its embeddings.
- 1:33:11
But of course, there's another set of hints that are really useful, and that's all the words that came before. So now we're gonna let all the words talk to each other to share their context, to say, "Oh, your moves," but your moves in a fast context.
- 1:33:22
That's gonna change and shift your recommendations. You can kind of think of this as kind of like a superposition of recommendations for what the next word is going to be.
- 1:33:30
And then you're probably not gonna get it right the first time, so we're gonna let you refine that prediction about twelve times. And then finally, you'll come out with a predicted embedding, and we just gotta turn that into whatever the next word is based on how close it is to our known dictionary of embeddings.
- 1:33:45
And that's essentially one way of looking at the model in whole, uh, despite all the complexity that we went through. [clears throat]
- 1:33:53
So we've been through a lot of different parts of the model at a very high level. Uh, it is totally natural to feel like your brain is full. What I often tell folks coming through this is, um, don't expect full mastery, but my metrics for success is that you get the sense that mastery is within your grasp.
- 1:34:11
There's nothing in here that was so complex you can't understand it. You can understand the whole model. Mastery is, uh, within your grasp, and we can turn what appears to be magic into machinery that you can understand.
- 1:34:25
Okay. Uh, before you go, last thing I'll just say is just like your favorite, you know, AI model, I get better from human feedback. So to incentivize you to fill out the survey and join the mailing list, uh, there's a link in the Discord channel.
- 1:34:39
If you fill it out, uh, and join the mailing list, I will send you the PDFs from today's workshop. [clears throat] And then, uh, if you visit Spreadsheets-are-all-you-need, you can join the mailing list.
- 1:34:50
Uh, there's a YouTube channel as well where I've got a bunch of other videos. Uh, and then I also have a Patreon I just launched, and I'm available for consulting, training, and implementation.
- 1:35:00
Uh, thank you. I hope you enjoyed the presentation and feel like now it's a little less like magic. [audience applauding]
- 1:35:13
We have time for questions. Go ahead.
- 1:35:24
First of all, um, wonderful presentation. I learned, uh, so much. Um-
- 1:35:29
Oh, thank you.
- 1:35:31
So I use, uh, Super Whisper on Mac. It's free.
- 1:35:34
Yeah.
- 1:35:34
And so I just wanted your expert opinion on, like, the way that I'm using AI is very much like a voice speech-to-text, and I just ramble, and I just try to give it as much information, and sometimes I'll reiterate what I think is really important.
- 1:35:50
Yeah.
- 1:35:50
What, what are your thoughts on that? Is, is it, is it a good approach? Are there ways I can im-improve on that?
- 1:35:56
Okay. So the number one thing, uh, I was gonna... I have a video I'm working on for this. Like,
- 1:36:03
the number one thing I'd say is you have to treat it scientifically. You, you can have theories about how the model works, but you don't really know till you test it.
- 1:36:11
This whole, like, the whole space is very empirical. I'll give you an example. So one of the common things in prompting they used to tell you is, like, say please and thank you, or say, "My grandma used to do this, I'm gonna lose my job," right?
- 1:36:25
And that, that legitimately used to work. But, uh, there was a great paper, the prompting report, uh, by Sander, and he, he went and studied, and they tested a bunch of models.
- 1:36:35
And they found, like, you know, it turns out with later models it didn't work. And then, uh, Ethan Mollick's team also did a recreation of a similar test, and they just tested a bunch of models with a bunch of prompts.
- 1:36:47
They tried it with polite words, and they put it on the... And they just said, "Okay, which one's better?" And they found that it wasn't really helpful. Um, that being said, uh, generally,
- 1:36:58
you... for, like, one-shot use cases, like, I'm just using it... Like, I use, uh, a Whisper, uh, tool myself all the time. I do exactly what you describe. I, I brain dump.
- 1:37:09
But then what I do is I go through and I look out-- I look through and I fix up things, like if there's grammar, or I repeated something, or I said something wrong.
- 1:37:17
And the way to think about why you wanna do that is- It's a somewhat subtle point, but when we go back to this, this diagram, this whole process is fixed.
- 1:37:29
Like, it's got a limited amount of compute. It can only do a certain amount of thinking. If I put a token in and I know how many tokens were in the prompt, I can predict how many FLOPS.
- 1:37:37
This is why, you know, when, like, a model like DeepSeek was trained, we know likely how many FLOPS were used because this thing isn't just like a program, it's like a, it's like a formula.
- 1:37:46
We-- it's a very long formula. It's very fixed. So if you make the prompt do more work, if the prompt is gonna make the model do more work, you kind of think of it like it loses some ability to do some other thinking.
- 1:37:59
Um, and so if you have spelling mistakes, if you, uh, say something slightly wrong, uh, that's different from normal, then it has to spend some sense of compute fixing that up.
- 1:38:11
This is less true with the reasoning models today 'cause the one thing the reasoning models can do is they can repeat this process more times on their own. So this isn't like a hard and fast rule, but for-- I would say if what you're doing is probably fine.
- 1:38:23
It's probably better that you put as much as you can that is relevant in. Like, if the choice is I put more stuff in, but I had the grammar wrong, but I had the relevant stuff in, that's better than if you didn't have it in there.
- 1:38:35
But if you're trying to engineer a prompt going into your model, I would spend time trying to optimize what its behavior is with some evals to make sure. Or at least benchmark what it is, and then when a new model comes out, you can see whether the new model changes things.
- 1:38:48
That's why evals are so important, 'cause this whole s- space is very empirical. Good question.
- 1:38:59
Any others? Okay. Well, have a great conference. Hopefully, this will help. Oh, sorry, there's one more. Yeah, go ahead.
- 1:39:08
Yeah. I wonder what your take is on the new mixture of experts models that have come to be very, very fine-grained expert when used like-
- 1:39:17
Yeah
- 1:39:17
... LLaMA model.
- 1:39:19
Um, uh, I wouldn't call it my take, but I'll give you the conventional take. Um, so the question was, what, what's your take on the mixture of experts models?
- 1:39:29
Um, there's like-- Let me take a step back. There are three or four things that when you come out of this, you know, workshop that you probably-- we don't cover that you should know about.
- 1:39:40
One of those is RoPE for embeddings. Another is RLHF, which I've talked a little bit about at the end. Uh, um, and then the other is reasoning models, which we just talked about, where the, the model can kind of run itself through.
- 1:39:51
And the last one is mixture of experts. It's probably, like, the biggest, uh, one of the biggest top four changes. Um, what we are trying to do with the mixture of expert is that, first of all, it only s- it's only here in the perceptron, which tends to dominate a lot of the calculation inside of a model.
- 1:40:08
Um, and what you're trying to do is get more knowledge, use more parameters without increasing the amount of compute. So what you do is you conceptually take this perceptron, and you break it into pieces.
- 1:40:21
And then you say, "Depending on what token comes in, I'm only gonna use a subset of my perceptron's thinking." And that way you can be more efficient with your compute and actually potentially your memory too.
- 1:40:30
You can charge your memory nicely per device if you want to do stuff like that. Um, so it's very-- it has a lot of advantages on paper, um, and we've had some really great models based on it.
- 1:40:41
Um, the challenge is training an MoE model is, is difficult, and so it's taken a while for, uh, some of the open source community to, to catch up in that implementation.
- 1:40:52
But, you know, definitely something of, of the future. Um, and by the way, MoE is actually fairly old. There are some, you know, much older models before ChatGPT that did MoE in other contexts.
- 1:41:03
Um, but that's the job MoE is trying to do. It's trying to cram more knowledge and more parameters while keeping the amount of compute used, uh, lower, and that seems to definitely have a benefit.
- 1:41:14
You can think of it as giving it more knowledge. Um, so yeah.
- 1:41:18
Does that answer your question? Okay. Thank you, [REDACTED:gender]. [outro music]