How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS

What do you need to understand a language model well enough to build on it? The apparent prerequisite list—calculus, linear algebra, a machine-learning course—can make the internals seem inaccessible. Ishan Anand’s workshop starts with a different challenge: take technology that looks like magic and expose its machinery using tools a web developer already knows.

His earlier project, Spreadsheets Are All You Need, implemented GPT-2 Small in pure Excel functions. One student, a CFO named Joe, came to the material through spreadsheet fluency rather than machine learning. The JavaScript workshop adapts that approach, compressing an eight-hour class into a planned two-hour session: ordinary programming and some familiarity with matrix multiplication are enough to begin. Python, React, Vue and advanced calculus are unnecessary for this walkthrough.

The working environment is GPT-2 in your browser, opened in desktop Chrome. The route through the model emphasizes its boundaries: tokenization and embeddings turn text into numerical input; the language head turns the final numerical representation into a token. Between them, attention and a multilayer perceptron transform the representations. Understanding those operations also supplies the foundation for explaining training and the later move from text completion to assistant behavior.

The model’s learned parameters arrive as CSV files: large tables of numbers, rather than an opaque executable. To load the workshop implementation:

1. Follow the page’s link to the GPT-2 Small CSV archive on GitHub.
2. Download and unzip the archive.
3. Select all the extracted parameter files and drag them into the loader.
4. Wait for the status indicating that all model parameters are loaded.

The loaded parameters occupy approximately 1.5 GB in IndexedDB browser storage. Chrome needs sufficient available disk capacity; this figure describes stored weights, not runtime RAM consumption. Once the files are loaded, the model runs locally, and Anand describes it as usable without an internet connection.

The interface resembles a notebook. One kind of cell contains executable JavaScript; another evaluates a formula and displays the result as a table. Matrices are ordinary two-dimensional JavaScript arrays. A formula can call a function such as getFinalTokens and reference promptToTokens, the DOM ID of an earlier table. The interface’s bracket notation resolves that table and passes its contents into the function. It is a small layer of table-reference syntax around JavaScript.

That makes the model available to the same debugging techniques used on a web page. Anand adds console.log(matches), reruns the tokenization cell and inspects its intermediate matches in Chrome DevTools. Then he inserts debugger and reruns the cell again: execution pauses inside the model, where variables and control flow can be inspected step by step. Removing the statement lets the remaining cells run normally. The useful property is inspectability: an unfamiliar model operation becomes a function whose inputs and outputs you can examine.

Give the model Mike is quick. He moves, and its immediate job is to predict the next token—here, quickly. It does not produce a paragraph in one operation. To continue generating, append the selected token to the input and run another prediction. Anand illustrates a subsequent continuation with and; repeating this feedback loop produces longer text or code. This is autoregressive generation. The browser workshop stops at one predicted token because that is the core operation it is exposing.

A sentence-completion problem becomes a numerical problem through a sequence of representations. First split the text into tokens. Then look up numerical vectors for those tokens, perform the model’s arithmetic, score possible output tokens and select one. Anand initially draws a simplified dictionary in which a word maps to a single number and an output is matched to nearby dictionary entries. That picture motivates the conversion, but the actual representations are vectors, and the output stage computes vocabulary scores. Randomness enters when a decoding procedure samples from those scores; greedy decoding instead selects the highest-scoring token.

GPT-2 dates to 2019, roughly three years before ChatGPT and four before GPT-4. Its release is often remembered through the phrase “too dangerous to release”; OpenAI’s announcement initially withheld the large model while releasing a smaller version. Its educational value is the continuity of its components with later transformers. Anand cites EleutherAI’s comparison with Llama 2 and uses a model family tree to make that point: learning this older, manageable implementation provides a foundation for recognizing the parts in newer systems.

A token is a vocabulary unit with an integer ID. It might be a whole word, part of a word, punctuation or a unit that includes a leading space. The running sentence happens to be easy to read as tokens, but shared spelling does not guarantee shared boundaries. When the network stalls, Anand switches to his local implementation and compares reindeer with reinjury. The latter includes a leading-space rein unit followed by jury; reindeer receives a different, three-token decomposition despite sharing the same initial letters. Tokenization is doing more than splitting strings at predictable word boundaries.

Why not give every dictionary word its own ID? Unknown words and misspellings immediately become a problem. An UNK token preserves the fact that something occurred but loses its identity. Unexpected languages and conventions matter too: Anand points to GPT-2’s rudimentary translation behavior and its ability to use TL;DR as a summarization cue. Retaining unfamiliar text makes such patterns available to training. A whole-word vocabulary also becomes large: his rough comparison is 170,000 English words against about 50,000 GPT-2 tokens, before adding other languages. More vocabulary entries require more parameters in the tables that represent and score tokens.

Character tokenization makes the opposite tradeoff. The example prompt occupies six rows in the model’s intermediate matrices, one per token. Turning every character into a token would make those matrices taller throughout processing. Individual characters also carry less immediate semantic information, leaving more structure for the model to learn. Anand uses the familiar jumbled-letter chain email as an analogy for recognizing larger units; its Cambridge attribution is part of the email, not a research result established here.

Subwords balance vocabulary size against sequence length.

Byte-pair encoding, or BPE, separates two jobs: learning a vocabulary from a corpus and applying its learned merge rules to new input. It began as a compression algorithm in the 1990s. The learning objective is easy to see in Anand’s small corpus of low, lower, newest and widest: repeated words retain their frequencies so that common text receives a more efficient representation. Dots indicate token boundaries, and an underscore marks a trailing space. This teaching convention differs from GPT-2’s leading-space units.

Begin with individual lowercase characters, then count adjacent pairs. In this corpus, e • s occurs nine times: six times in newest and three times in widest. The frequency table identifies it as a most-frequent pair, so it becomes the next merged vocabulary unit.

The procedure then repeats:

1. Add es to the vocabulary and replace each adjacent e • s occurrence with it.
2. Count adjacent pairs again, now treating es as one unit.
3. Merge es • t, which also occurs nine times, into est.
4. Continue selecting frequent pairs and rewriting the corpus.

For example, the affected rows progress from n • e • w • e • s • t • _ to n • e • w • es • t • _, then to n • e • w • est • _. After ten passes in the demonstration, frequent words such as low and newest have become whole tokens. The vocabulary has grown while the number of tokens needed to represent the corpus has shrunk. Units such as low and est resemble meaningful language fragments, but the algorithm reached them by frequency counting, without understanding their meanings.

At inference time, the tokenizer uses the vocabulary learned during training. The browser implementation first applies OpenAI’s regular expression to split the prompt into preliminary chunks. In Mike is quick. He moves, punctuation separates out, while spaces remain attached to chunks such as He and moves. These chunks are not yet the final tokens. The implementation then loads vocab.bpe, which supplies the learned ordering of merges. Its display adds rank and score columns; early entries include space-plus-t and space-plus-a. A separate map connects resulting tokens to IDs.

The table for quick makes the operation visible. Start with its characters, inspect adjacent pairs and select the available merge with the best learned rank. In the displayed sequence, i and c combine first; later merges form qu and ick, eventually producing the full token. Inference reuses the learned ordering rather than building a new frequency ranking from the prompt. The unusually expansive table layout mirrors the Excel version so readers can follow the same intermediate states in either implementation.

This representation can complicate character-level questions. Counting the r characters in strawberry requires recovering spelling information from tokens rather than reading a sequence of individual letters. In Riley Goodside’s example, capitalization, leading spaces and quotation marks produce six different token patterns for variants a person readily recognizes as the same word. Tokenization can worsen these errors without being their sole cause.

The preference for subwords is empirical, not a requirement of all models: research systems also explore word and character representations. Nor must a token represent text. A Vision Transformer can use image patches as its units, while Anand cites Waymo’s use of spatial trajectories. The general question is what units make the input useful for the computation that follows.

A token ID does not itself describe meaning. GPT-2 Small maps every token—including a period—to a vector of 768 numbers. In the browser, each token becomes a row with the same width. Anand notes an implementation-specific limitation when experimenting with other prompts: the workshop importer can mishandle foreign-language characters because of a character mismatch during import.

Think of a housing table. A street address identifies a house, while square footage, bedrooms, bathrooms and price describe its properties. The token ID acts like the address; the embedding acts like the descriptive columns. A useful embedding places related tokens in related parts of a high-dimensional space. In Anand’s two-dimensional “word island,” happy and glad sit together, dog and cat occupy another region, and sad belongs near the emotions even though it differs from happiness.

A deliberately simple coordinate system shows why vectors are useful. Give the first coordinate the label authority and the second gender, with these teaching values:

Then ordinary coordinate arithmetic produces a relationship:

The analogy becomes a mathematical operation: king relates to man as queen relates to woman.

word2vec made such relationships famous, including country–capital, profession and food associations. Its results were imperfect, but examples such as France–Paris, Japan–Tokyo and Italy–Rome illustrate a useful idea: a recurring direction through vector space can represent a relationship, suggesting Canada–Ottawa through a similar offset. Actual learned embeddings have hundreds of dimensions without the toy table’s straightforward labels. Even so, unlabeled columns can support comparison—just as similar numerical house records remain recognizable after removing their column headings.

Where do the embedding values come from? Training begins with random parameters, including random embeddings. Take an observed passage, Mike is quick. He moves quickly., withhold the continuation and ask the model to predict it. A randomly initialized model might produce something like haircut. Comparing its prediction with the observed quickly supplies a training signal. Backpropagation computes how parameters affect the error, and an optimizer uses those gradients to adjust them. Repeating this across many passages lets the model learn patterns of grammar, names and facts without individually hand-labeling those concepts.

The statistical intuition is that ice often appears with cold, while steam appears with hot. Even without knowing the words beforehand, an observer could infer relationships from their contexts. This is the distributional hypothesis, captured by the phrase “You shall know a word by the company it keeps.” Words with related uses tend to occur in related contexts.

In the longer class, Anand builds primitive embeddings from Wikipedia using a simplified, GloVe-inspired method. Count which words occur within a window—three words to either side in the example—and record those counts in a word-by-word matrix. An example pair might occur together five times. A compact embedding can then be imagined as retaining useful structure from that enormous table in far fewer columns. This compressed co-occurrence picture explains why embeddings contain contextual information; GPT-2 itself learns its token table jointly through next-token training rather than literally compressing that count matrix.

Why compare embeddings using cosine similarity rather than straight-line distance? Cosine compares vector directions: aligned vectors score 1, orthogonal vectors score 0 and opposite directions score −1. Semantic opposites need not point in opposite directions. Happy and sad, for example, share emotional contexts. Anand reports that few pairwise similarities among GPT-2 token embeddings are negative, and those tend to lie near zero.

The co-occurrence analogy makes direction meaningful. Consider three words counted against the same two context words:

The first two vectors, (10, 10) and (50, 50), have the same relative pattern, though the second word is five times as frequent. They point in the same direction. (5, 20) lies closer to (10, 10) in Euclidean distance but expresses a different contextual proportion. Comparing direction separates that proportion from raw frequency.

Using an embedding is much simpler than learning one. GPT-2 Small’s model_wte table has 50,257 rows and 768 columns. Each token ID selects one row. In the running prompt, the leading-space token is has ID 318; the spreadsheet-style interface displays that zero-indexed row as row 319. The matching decimals in the source table and prompt table demonstrate a lookup, not a new calculation.

function tokenEmbeddings(tokenIds, model_wte) {
  return tokenIds.map(id => model_wte[id].slice());
}

The resulting matrix has one row per input token, preserving their order.

Embeddings also connect different modalities. CLIP learns relationships between images and text: repeated pairings of dog images with descriptions involving dogs can place the two representations into a useful relationship. The same general idea—learning comparable representations from associated observations—extends beyond word-to-word similarity.

The dog chases the cat and The cat chases the dog contain the same words but describe different events. Token identity alone does not encode that difference. Some operations that combine numerical representations do not preserve order by themselves, so the model needs an explicit position signal. Anand illustrates this as offsetting a token such as woman according to where it appears in the prompt: related representations retain the token’s identity while carrying location information.

The original Transformer used sinusoidal position encodings. Sine and cosine oscillate between −1 and 1; the illustration represents that bounded variation as a small cloud around a token’s original position. GPT-2 instead learns a position table during training. The geometry is an intuition for adding position information, not a claim that each real embedding follows a literal two-dimensional circle.

GPT-2 Small’s model_wpe table has 1,024 position rows and 768 columns. Its rows correspond to the available positions in the model’s context. For each token, select its token vector and add the vector for its position, element by element:

function embedPrompt(tokenIds, model_wte, model_wpe) {
  if (tokenIds.length > model_wpe.length) {
    throw new RangeError("Prompt exceeds the position table");
  }

  return tokenIds.map((tokenId, position) =>
    model_wte[tokenId].map(
      (value, dimension) => value + model_wpe[position][dimension]
    )
  );
}

Only the position rows needed by the actual prompt are used. The vocabulary table is indexed by token ID; the position table is indexed by location.

A plot of happy at different positions shows the resulting offsets; labels such as happy 3 identify positions, not new vocabulary entries. Many newer models use RoPE, or rotary position embeddings, instead of GPT-2’s additive learned table. Asked how GPT-2 obtains its position values, Anand returns to training: make those values learnable parameters, then propagate the prediction error through them along with the other parameters.

The representations now enter transformer blocks, commonly called layers. Anand uses “block” to avoid confusing a transformer layer with the layers inside its neural network. The implementation’s numbered steps preserve the layout of the Excel teaching material; attention occupies steps 4–9. The two main operations inside each block are multi-head attention and the multilayer perceptron.

Attention lets a token’s representation incorporate relevant information from other tokens. In the running sentence, He can use the earlier Mike to resolve its reference. Quick has several senses—physical speed, intelligence, the sensitive part of a fingernail, or alive in “the quick and the dead.” By the time the model reaches moves, the earlier quick helps make fast physical movement a useful interpretation for predicting the continuation.

Anand imagines tokens exerting a kind of gravity in embedding space: relevance determines which interactions matter, while values supply the information that changes the representation. In OpenAI’s GPT-2 implementation, the actual mechanism uses scaled query–key products, a causal mask, softmax and a weighted combination of value vectors. The metaphor’s useful consequence is contextual movement: the initial embedding for moves must accommodate both fast and slow uses, but attention to quick can shift the current hidden representation toward fast movement. It does not rewrite the stored vocabulary embedding.

Step 7 exposes an attention matrix, with prompt tokens on both axes. Each row describes how one position distributes attention over available positions. Further heads appear elsewhere in the wide table; Anand navigates to them by scrolling 64 spaces. The upper triangle is zero because GPT-2 is causal: a position can attend to itself and earlier positions, never later ones. Each row sums to one. In the displayed first head of the last block, moves assigns approximately 16% of its attention to Mike and 23% to is. Those values describe that particular head, block and prompt.

The second major operation in a block is the multilayer perceptron, or MLP. Artificial neural networks borrow inspiration from biological neurons without directly simulating a brain. The introductory analogy is activation: incoming signals may or may not cause a neuron to transmit a signal onward. The computational version multiplies inputs by weights, sums them, adds a bias and applies an activation function.

For one neuron, the operation is:

A simple activation is ReLU: negative inputs become zero, while positive inputs pass through unchanged.

function reluNeuron(inputs, weights, bias) {
  const total = inputs.reduce(
    (sum, value, index) => sum + value * weights[index],
    bias
  );
  return Math.max(0, total);
}

Changing the weights and bias changes which inputs activate the neuron and how strongly.

An MLP arranges neurons into layers, with each neuron connected to all the outputs of the preceding layer. Layers between input and output are hidden layers. You will also encounter the terms fully connected network and feed-forward network; they overlap with MLP without being interchangeable in every context. Many weighted sums can be written together as matrix multiplication plus a bias, followed by activation. This is why the implementation repeatedly performs matrix multiply and add operations. Anand’s interactive matrix-multiplication example connects the compact matrix notation back to the individual neuron calculations.

The attraction of an MLP is its ability to approximate functions. Start with a simple target, a parabola, and a network with one input, one output and two hidden ReLU neurons. One ReLU supplies a line segment for the right side; a flipped one supplies the left side. Adding them gives a rough piecewise-linear approximation over the displayed interval.

In the interactive demo, changing parameters moves those pieces while a mean squared error display tracks the mismatch. More neurons supply more pieces: Anand shows approximations using 8, 20 and 200 neurons. In the 200-neuron illustration, the approximation is visually difficult to distinguish from the parabola at the displayed scale. The universal approximation theorem motivates this capacity under suitable assumptions; it does not guarantee that a chosen training procedure will find the desired parameters.

For language modeling, the unknown function maps context to useful predictions. Training examples supply observed continuations, so parameters can be adjusted without a programmer writing the language rules or manually turning every knob. The distinction matters: backpropagation calculates gradients; an optimizer updates parameters using them. Anand’s foggy-mountain analogy describes the update process. The hiker’s position represents the parameters, elevation represents error, and the local slope suggests a direction that decreases error. The slope does not reveal the whole landscape or promise the best possible minimum. Anand also warns, from experience, that following downhill terrain is not a reliable real-world hiking strategy.

In GPT-2 Small, the MLP maps a 768-dimensional input through a 3,072-dimensional hidden layer and back to 768 dimensions. The fourfold expansion is an empirical architectural choice. Steps 13–15 expose the three operations:

Unlike the introductory ReLU example, GPT-2 uses GELU. The same MLP is applied independently to each token position, normally in parallel. Its output updates a hidden representation through the block’s residual connection; it is not an independently decoded next token.

The code is a direct composition of the earlier arithmetic. With row-oriented input representations X, the expansion multiplies the previous step’s output by mlp_fc_weights and adds a learned bias. GELU transforms the result, then another learned weight matrix and bias project it back:

The two weight matrices perform different jobs and are learned separately.

Training reaches much further than the MLP. Gradients also flow through token embeddings, GPT-2’s learned position embeddings, attention’s query/key/value parameters and learned LayerNorm parameters. Anand compares the process to a chef reconstructing a dish: the ingredients are the input, the desired dish is the target, and the architecture prescribes available operations. Optimization determines the quantities that make those operations produce a useful result.

GPT-2 Small has 12 transformer blocks, each using the same kinds of operations with its own learned parameters. These successive transformations refine the hidden representations before final vocabulary scoring. In the parameter names, h0 denotes the first block and h11 the twelfth. The educational interface reuses its displayed block machinery by retrieving DOM formulas, replacing the block identifier and rerunning the steps with the appropriate weights. Repeated structure does not mean shared weights.

After the last block, a final LayerNorm prepares the hidden representations for the language head. For next-token generation, use the representation at the final input position. It still contains 768 numbers, so it must be converted into scores over the vocabulary. GPT-2 reuses model_wte: each vocabulary row is dotted with that final vector. A 50,257-by-768 table multiplied by a 768-element column vector yields 50,257 scores, one per token. These scores are logits. They are dot-product compatibility scores, not cosine similarities or probabilities.

Softmax converts the logits into a probability distribution. Subtracting the largest score before exponentiation keeps the arithmetic numerically stable without changing the resulting probabilities:

function vocabularyLogits(finalHidden, model_wte) {
  return model_wte.map(embedding =>
    embedding.reduce(
      (score, value, index) => score + value * finalHidden[index],
      0
    )
  );
}

function softmax(logits) {
  const maximum = logits.reduce((a, b) => Math.max(a, b), -Infinity);
  const weights = logits.map(logit => Math.exp(logit - maximum));
  const total = weights.reduce((sum, weight) => sum + weight, 0);
  return weights.map(weight => weight / total);
}

The returned entries are nonnegative and sum to one.

The browser’s predicted-token cell takes a simpler path: it selects the maximum logit directly. Softmax preserves the ordering, so computing probabilities is unnecessary for this greedy choice. Anand uses deterministic decoding to make the output comparable with OpenAI’s code or Hugging Face Transformers. He submits the running prompt to the latter during the workshop, then leaves the request to finish while the network is slow.

Other decoding rules control which tokens can be sampled:

Anand illustrates top-k with ten candidates and top-p with thresholds such as 80% or 90%. After restricting the candidates, renormalize their probabilities before sampling. These rules reduce the chance of drawing an extremely unlikely continuation such as haircut.

When the delayed Hugging Face result returns, it agrees with the browser example: Mike is quick. He moves continues with quickly. The displayed winning token is ID 2952, with a maximum logit of approximately −129.44. The spreadsheet shows it on row 2953 because its visible row numbering starts at one. A negative score can win: the decision depends on being larger than the other logits, not on being positive. Looking up ID 2952 in the token dictionary completes the return from numbers to text.

The move from GPT-2 to ChatGPT involved scale and architectural changes, but a central distinction is the behavior training asks the model to produce. A base model learns to continue internet text. Given First name:, GPT-2 produces form-like continuations involving passwords and email. Given Hello class, it heads toward code containing Foo and public static void main, rather than replying as a student or teacher. Those continuations make sense as text patterns even when they are unhelpful as conversation.

The same model can answer that France’s capital is Paris, so useful knowledge is mixed with other continuation patterns. Assistant training tries to make the desired behavior more dependable. Anand introduces a four-stage progression, starting with internet pretraining and then supervised training on examples of helpful responses. He describes a general range of 10,000–100,000 contractor-written prompt/response examples for that assistant-training stage.

The Stanford Alpaca dataset makes the instruction/response format easy to inspect as JSON. Anand shows examples asking for healthy-living tips and primary colors. Alpaca is a format illustration here: its outputs were generated by text-davinci-003, rather than being the contractor-written examples just described. Supervised fine-tuning teaches the model to imitate this kind of desired response, but imitation alone does not capture every preference about helpfulness and safety. That motivates reinforcement learning from human feedback.

Imagine an agent navigating a maze with monsters, obstacles, power-ups and a goal. It tries routes, receives scores and learns a policy that favors better outcomes. Unlike copying a demonstrated route, reinforcement learning can search for behavior based on its reward. A generated sentence can likewise be viewed as a path through successive token choices. But scoring language is more subtle than counting desirable or undesirable words: not angry cannot be judged by treating angry in isolation.

To obtain a scoring function, collect preferences between responses to the same prompt. Anand shows examples from Anthropic’s helpful and harmless preference data. For a pumpkin-pie request, the preferred response supplies a recipe, while the rejected one tells the user to buy pumpkin and read the package. A harmlessness example prefers constructive guidance about stress-related drinking over encouragement to drink without restraint. Such pairs express judgments about the response as a whole.

The distinction between the last two stages is crucial:

1. Pretraining: learn broad text patterns from internet passages.
2. Supervised fine-tuning: imitate examples of desired assistant responses.
3. Reward modeling: use chosen/rejected response pairs to train a separate scoring model.
4. Reinforcement learning: update the assistant to favor responses that score well under that reward model.

At stage three, learning the scorer has not yet updated the assistant itself. Stage four uses that scorer to change the assistant’s behavior. Anand closes this part by pointing to R1-Zero and GRPO as developments making reinforcement learning especially relevant beyond the earlier assistant-training recipe.

The whole pipeline can also be understood through recommendation systems. Tokenization finds an efficient representation of the input. Embeddings then capture relationships, much as representations of books, films or music can group related items from user behavior. Anand shows music-genre clustering as an example. A word’s contextual relationships provide useful clues about plausible continuations, just as an item’s relationships help recommend another item.

The neural network learns to turn those clues into predictions. Attention adds the current passage’s evidence: moves in the presence of an earlier quick should support different continuations from moves in a slow context. Successive blocks refine that representation, and the language head scores the vocabulary. The goal of walking through the implementation is not instant mastery of every operation. It is recognizing that each operation is understandable, inspectable machinery.

For continued study, Anand directs attendees to the workshop’s Discord survey and mailing-list signup for the PDFs, along with the project’s mailing list and YouTube channel. He also mentions Patreon, consulting and training. The closing questions then return to two practical consequences of this foundation: how to evaluate prompts and how newer models increase capacity without activating every parameter.

An audience member describes using superwhisper on a Mac to dictate long prompts, ramble through relevant information and repeat important points. Is that a good approach? Anand’s first answer is empirical: theories about model behavior need tests. He cites The Prompt Report and work by Ethan Mollick’s team while discussing how the usefulness of politeness and emotional appeals can change across models. His practical lesson is to avoid treating a remembered prompting trick as a permanent rule.

For his own one-off use, Anand also dictates a brain dump, then corrects mistakes, repetition and awkward grammar. His intuition is that a fixed forward pass has bounded computation: making the model decipher noisy input may make the requested task harder. That is a heuristic about input quality, not a measured exchange rate between spelling errors and reasoning capacity. Reasoning models qualify the picture because they can use additional generated steps. Relevant information is still more valuable than omitting useful context merely to make the prompt cleaner.

For an engineered prompt, establish an evaluation baseline and measure changes against it. Keep representative tasks, compare prompt variants and rerun the evaluation when the model changes. Understanding the architecture suggests hypotheses; evals tell you whether those hypotheses improve the behavior you actually need.

The final question asks about increasingly fine-grained mixture-of-experts models, mentioning Llama. Anand places MoE alongside RoPE, RLHF and reasoning models as major extensions to investigate after learning the GPT-2 foundation. In the architecture he describes, MoE changes the MLP portion of the transformer, which accounts for a substantial share of its computation.

Instead of sending every token through all the available feed-forward capacity, route each token to a subset of experts. The model can then have more total parameters while activating only part of them for a particular token. Expert weights can also be distributed across devices. That deployment arrangement can help manage memory, but inactive experts do not cease to require storage. The distinction is between total parameter capacity and the computation active on each token.

The difficulty shifts toward training and coordination. Anand identifies training complexity as one reason open implementations took time to mature, and notes that MoE predates ChatGPT. Its purpose is a concrete extension of the machinery already examined: retain more learned capacity while controlling how much of it must run for each prediction.