Training an LLM from Scratch, Locally

Can you train a language model on a laptop without downloading someone else’s weights? That is the starting point of Angelos Perivolaropoulos’s workshop: use PyTorch and a few basic libraries to build the model, train it, and generate text. Perivolaropoulos introduces himself as the lead of ElevenLabs’ speech-to-text team, working on real-time transcription for agents. He also introduces Scribe v2 and its public-benchmark leadership claim; ElevenLabs’ announcement makes that claim for transcription word error rate and distinguishes the batch model from Scribe v2 Realtime.

The exercise exposes the core machinery while leaving large-scale optimization and specialization for later. The workshop repository offers two routes: train locally, with roughly 16 GB of memory as the suggested starting point, or use Google Colab’s GPU runtime. The model is deliberately small enough that acquiring a cluster is not the first problem to solve.

Karpathy’s nanoGPT is the motivating reference: a compact PyTorch project that makes language-model training approachable. This workshop uses a small GPT-2-style causal decoder. Its workflow has four pieces: choose a tokenizer, assemble the architecture, write the training loop, and implement generation. A data-limited exercise favors a small vocabulary and small embeddings; the decoder then combines causal self-attention, feedforward layers, and normalization. Longer contexts and greater training throughput introduce additional engineering constraints.

Perivolaropoulos emphasizes training and post-training as major sources of capability gains, citing successive GPT and Gemini releases. His comparison between Gemini 3 and Gemini 3.1 concerns improvements on particular benchmarks, not a doubling of overall capability, and does not establish which training changes caused them. For this exercise, the immediate goal is simpler: make a small model learn, then run its inference on ordinary hardware.

Use Python 3.12 and choose the execution path that matches your machine:

• Local: install uv, open the repository, and run uv sync to create the environment and install dependencies. The workshop uses a scratchpad for assembling code.
• Colab: create a notebook and install torch, numpy, tqdm, and tiktoken. The character tokenizer itself does not require tiktoken; it is included for experimentation.
• Device: the code supports Apple Silicon through mps, NVIDIA GPUs through cuda, and CPU execution. Memory availability constrains batch size, while the device affects training speed.

uv sync

For the notebook dependency cell:

%pip install torch numpy tqdm tiktoken

The first design decision is how to represent the data. Perivolaropoulos illustrates its importance with a TTS planning example: a team might spend six months considering the tokenizer and two considering the architecture. A language model processes vectors, so text must first become integer IDs and then learned embeddings. The Shakespeare corpus used here contains 65 distinct characters, making a character-level tokenizer a small, transparent starting point.

Build a sorted character vocabulary, assign each character an integer, and invert the mapping for decoding. stoi is simply a string-to-integer dictionary, not a separate library. The corpus is supplied with the project; the mapping must remain the same when generating from a saved model.

class CharacterTokenizer:
    def __init__(self, text: str):
        self.chars = sorted(set(text))
        self.stoi = {ch: i for i, ch in enumerate(self.chars)}
        self.itos = {i: ch for ch, i in self.stoi.items()}

    def encode(self, text: str) -> list[int]:
        return [self.stoi[ch] for ch in text]

    def decode(self, ids: list[int]) -> str:
        return "".join(self.itos[i] for i in ids)

The embedding layer later turns these IDs into trainable vectors. An unfamiliar character is outside this particular tokenizer’s vocabulary; unlike a byte-based tokenizer, this dictionary has no universal byte fallback.

With 65 characters, there are 65² = 4,225 possible ordered bigrams. That small space motivates the choice: repeated adjacent-character patterns are easier to encounter in a small corpus than combinations from a huge vocabulary. Perivolaropoulos extends this into a vocabulary-squared heuristic for data requirements, using a 200,000-token vocabulary as the contrast. Treat that as intuition about sparsity, not a minimum-data formula: natural language does not require every possible token pair to occur. The practical warning is that a large off-the-shelf vocabulary is a poor match for this tiny training budget.

Character tokenization moves work into the transformer. In the sentence The sky is blue, a model using word-sized pieces can learn a relationship between sky and blue directly. A character model must first compose s, k, y and b, l, u, e into useful representations. It also processes and generates more tokens for the same text. These costs explain the workshop’s trade-off without requiring the stronger claim that character models can never learn well.

Byte-pair encoding, or BPE, learns reusable pieces by merging frequent adjacent patterns in training data. It offers a path beyond the exercise when more data and compute are available. Vocabulary size also changes the embedding table directly: each vocabulary entry needs a vector of the chosen width.

These counts are just vocabulary size multiplied by embedding width. The larger table alone would exceed the size of the workshop’s small model, before counting attention or feedforward layers.

The architecture is approachable because a few components repeat. Perivolaropoulos encourages implementing them first and deepening the mathematical understanding through experiments. Many newer systems retain these components while changing attention efficiency, context handling, or other details; hybrid architectures can depart further from this baseline.

• Multi-head self-attention lets each position combine information from earlier positions. Different heads can learn different relationships. Supporting a much longer context requires managing the resulting computational and memory costs.
• The MLP, also called the feedforward network, transforms each position’s contextual representation into features that subsequent layers and the output head can use.
• Residual connections add a learned update to the incoming representation. A layer can refine what is already present rather than replacing it wholesale.
• Layer normalization regulates activation scale. Perivolaropoulos illustrates the problem with repeated tenfold amplification: an initially small value could become enormous after several layers.

Together these components make repeated processing trainable; they do not assign a manually specified meaning to each neuron or head.

An audience question makes the tokenizer trade-off concrete: Python has common keywords, but programmers invent variable and function names constantly. A learned tokenizer may encode frequent pieces such as for, enumerate, foo, or bar compactly. An unusual identifier instead breaks into smaller known pieces, potentially down to bytes. It does not need a dedicated vocabulary entry to be representable. The corpus determines which patterns become compact; poorly represented languages or unusual naming conventions can therefore consume more tokens.

The model configuration separates vocabulary size, context length, depth, attention structure, and representation width. The workshop starts with a vocabulary of 65, a context of 256 tokens, six layers, and width 384. Here, 384 is the exercise’s embedding width, not a general specification for GPT-2. Heads may learn patterns associated with punctuation or grammar, but those roles are examples of possible learned behavior, not labels assigned in code. A top-level GPT module receives this configuration and organizes its submodules, using PyTorch’s ModuleDict in the demonstration.

Context length and parameter capacity are different controls. A small model can process long sequences, and a large model can process short ones. Full causal attention can look across the supplied past; windowed attention restricts how far back a position looks. This exercise trains on sequences no longer than 256 tokens, so it cannot simply be asked to handle arbitrary longer inputs.

Asked which settings to increase for more capacity, Perivolaropoulos tentatively suggests increasing depth and heads, while expressing uncertainty about width. There is no single scale-up knob: increasing the number of heads at fixed width does not by itself proportionally increase the projection parameters. A short character context also loses earlier material quickly. Jumping from 256 tokens to millions would require addressing memory, attention cost, positional representation, and training stability—not just editing a configuration value.

The top-level model has two embedding tables: one for token identity and one for position. Their outputs are added, passed through the transformer blocks, normalized, and projected through the language-model head. The head produces logits—unnormalized scores for the vocabulary. During generation those scores become a probability distribution for choosing the next token.

The forward method below expresses that flow. Its input has shape (batch, time), hidden representations have shape (batch, time, width), and logits have shape (batch, time, vocab_size). When targets are supplied, cross-entropy evaluates the prediction at every position; without targets, the method returns scores for inference.

import torch
import torch.nn.functional as F


def forward(self, idx, targets=None):
    _, time = idx.shape
    if time > self.block_size:
        raise ValueError("Input exceeds the model context length")

    positions = torch.arange(time, device=idx.device)
    x = self.token_embedding(idx) + self.position_embedding(positions)
    for block in self.blocks:
        x = block(x)
    logits = self.lm_head(self.final_norm(x))

    loss = None
    if targets is not None:
        loss = F.cross_entropy(
            logits.reshape(-1, logits.size(-1)),
            targets.reshape(-1),
        )
    return logits, loss

For the workshop vocabulary, the final dimension contains one score for each of the 65 characters.

Attention learns which earlier representations matter at the current position. The sky–blue example returns here: with character tokens, the model must compose the letters as well as learn the relationship. Multi-head attention projects representations into queries, keys, and values, processes heads separately, then combines their outputs through a projection. Causal masking prevents a training position from seeing the future token it is supposed to predict.

The MLP then transforms the attention-enriched representation. An audience question asks whether all transformer blocks share the same MLP weights. They do not: each block has its own attention, normalization, and MLP parameters, even when every block uses the same architecture. Separate module instances create separate learned transformations at successive depths.

This model uses a pre-normalization arrangement: normalize before attention, add the attention update, normalize again, and add the MLP update. The residual is visible directly in the two additions:

from torch import nn


class Block(nn.Module):
    def __init__(self, width, heads):
        super().__init__()
        self.ln1 = nn.LayerNorm(width)
        self.attention = nn.MultiheadAttention(
            width, heads, batch_first=True
        )
        self.ln2 = nn.LayerNorm(width)
        self.mlp = nn.Sequential(
            nn.Linear(width, 4 * width),
            nn.GELU(),
            nn.Linear(4 * width, width),
        )

    def forward(self, x):
        time = x.size(1)
        mask = torch.ones(
            time, time, device=x.device, dtype=torch.bool
        ).triu(1)
        normalized = self.ln1(x)
        update, _ = self.attention(
            normalized, normalized, normalized,
            attn_mask=mask, need_weights=False,
        )
        x = x + update
        x = x + self.mlp(self.ln2(x))
        return x

The key distinction is x = x + update, rather than replacing x with update. Perivolaropoulos does not recall the historical residual implementation in the particular Karpathy resource being discussed; the connection in this model is explicit.

Collect the architecture into model.py. The most reliable parameter count comes from the instantiated model:

def parameter_count(model):
    return sum(parameter.numel() for parameter in model.parameters())

The workshop describes the default model as approximately 10 million parameters. Most of those parameters live in the repeated transformer blocks, rather than in the small character embedding table.

Attention has query, key, value, and output projections. The MLP expands the width from 384 to 1,536 and projects back. Together, attention and MLP account for roughly 1.8 million weights per block; six such blocks explain the roughly 10-million-parameter scale. The spoken description briefly calls 1.8 million the total, but that figure fits one block. Biases, normalization parameters, the output head, and any weight sharing determine the exact model total.

The training task needs to be both learnable and easy to recognize when it starts working. Shakespeare-like verse meets that requirement better than competitive programming: the progression from random characters to plausible words, names, and dialogue is visible in samples. The objective is next-token prediction. For each input position, the target is the following token.

Take a short teaching sequence, KING. Using a temporary vocabulary G: 0, I: 1, K: 2, N: 3, its IDs are [2, 1, 3, 0]. Shift the sequence to create inputs [2, 1, 3] and targets [1, 3, 0]: predict I after K, N after KI, and G after KIN. The real tokenizer uses the full corpus vocabulary; the shift operation is the same. Causal attention keeps each prediction from seeing later input positions.

The supplied corpus contains about one million characters. Split it into training and validation portions, then sample contiguous windows without shuffling the characters inside them. Each batch contains 64 sequences of 256 input tokens, with targets shifted one position forward. The intentionally simple loader can be expressed as follows:

def get_batch(data, batch_size=64, block_size=256, device="cpu"):
    starts = torch.randint(len(data) - block_size, (batch_size,))
    offsets = torch.arange(block_size)
    positions = starts[:, None] + offsets[None, :]
    inputs = data[positions]
    targets = data[positions + 1]
    return inputs.to(device), targets.to(device)


def choose_device():
    if torch.cuda.is_available():
        return "cuda"
    if torch.backends.mps.is_available():
        return "mps"
    return "cpu"

Call get_batch on the chosen split’s one-dimensional integer tensor. CUDA and MPS accelerate the computation where available; CPU remains a supported, slower route.

The learning rate controls the scale of weight updates. Too large a rate can overshoot useful regions and destabilize training. The workshop uses a 100-step warm-up, followed by cosine decay through a 5,000-step run. Perivolaropoulos prefers a nonzero final rate so training can continue with meaningful updates. Learning-rate decay and weight decay are separate mechanisms: the former changes update size over time; AdamW’s weight decay regularizes weights. A scheduler or explicit function supplies the cosine schedule—AdamW does not create it automatically.

Keep that separation explicit in code. This schedule accepts the peak and final learning rates as arguments:

import math


def learning_rate(step, peak, floor, warmup=100, total=5000):
    if step < warmup:
        return peak * (step + 1) / warmup
    progress = min((step - warmup) / (total - warmup), 1.0)
    cosine = 0.5 * (1.0 + math.cos(math.pi * progress))
    return floor + cosine * (peak - floor)

Initialize the six-layer, six-head model with width 384 and context 256, then initialize its optimizer. tqdm makes the steps and losses visible. Training loss measures fit to sampled training data; validation loss measures predictions on held-out text. If the first keeps falling while the second rises, the model is fitting the training set at the expense of generalization.

Each optimization step updates the learning rate, clears old gradients, computes loss, backpropagates, and applies the optimizer update:

def train_step(model, optimizer, inputs, targets, lr):
    model.train()
    for group in optimizer.param_groups:
        group["lr"] = lr
    optimizer.zero_grad(set_to_none=True)
    _, loss = model(inputs, targets)
    loss.backward()
    optimizer.step()
    return loss.item()


@torch.no_grad()
def validation_loss(model, batches):
    was_training = model.training
    model.eval()
    losses = [model(x, y)[1].item() for x, y in batches]
    model.train(was_training)
    return sum(losses) / len(losses)

The workshop saves checkpoints every 1,000 steps and generates samples along the way. For a resumable implementation, preserve optimizer and step state alongside model weights; also retain the configuration and vocabulary needed to reconstruct the model. Samples provide a qualitative view that a single loss value cannot.

A uniform predictor over 65 characters has cross-entropy ln(65) ≈ 4.17 nats per token. That gives the randomly initialized model a useful reference point: before learning, it has little basis for favoring the correct character. As training progresses, Perivolaropoulos describes the following approximate milestones for this Shakespeare exercise.

These are task-specific observations for interpreting samples, not stopping thresholds to transfer to a different tokenizer or corpus. The held-out trajectory matters more than crossing a particular number.

In Perivolaropoulos’s test, 200 steps produced nonsense with validation loss around 3.5; output improved around 800 steps and again around 1,000. He reports the best point at roughly 2,400 steps, after which validation loss rose and output became less creative. Validation loss is a cheap signal for this exercise; a more serious training effort should also measure task performance. The immediate workshop milestone is still to assemble the pieces and get a first working run.

Generation repeatedly runs the model and appends one chosen token. Greedy decoding always chooses the highest-scoring token: in Perivolaropoulos’s example, a token with 80% probability wins over one with 15% every time. He prefers sampling for creative verse and contrasts that with transcription, where unexpected creativity is undesirable. These are task preferences; greedy decoding remains a valid choice when deterministic selection is wanted.

Temperature changes the sharpness of the distribution before sampling. Lower positive values concentrate probability on higher-scoring tokens; higher values spread it more broadly. Sampling can produce variety, but also repetition or unwanted stopping tokens in models that have them. Perivolaropoulos suggests 0.7 as a starting point. Top-k further restricts sampling to the highest-scoring candidates: his example keeps five plausible tokens and excludes an unlikely sixth.

The order is logits, temperature scaling, optional candidate filtering, softmax, then sampling. Only the final position’s logits choose the next token. Keep the input within the trained context window as the output grows:

@torch.no_grad()
def generate(model, ids, new_tokens, block_size,
             temperature=0.7, top_k=None):
    if temperature <= 0:
        raise ValueError("Use a positive temperature for sampling")
    model.eval()
    for _ in range(new_tokens):
        logits, _ = model(ids[:, -block_size:])
        scores = logits[:, -1, :] / temperature
        if top_k is not None:
            if top_k < 1:
                raise ValueError("top_k must be positive")
            k = min(top_k, scores.size(-1))
            values, indices = torch.topk(scores, k)
            filtered = torch.full_like(scores, float("-inf"))
            scores = filtered.scatter(1, indices, values)
        probabilities = torch.softmax(scores, dim=-1)
        next_id = torch.multinomial(probabilities, num_samples=1)
        ids = torch.cat((ids, next_id), dim=1)
    return ids

A fixed random seed makes it possible to repeat the sampling sequence in a controlled setup, which becomes useful for the workshop’s verse competition.

The project resolves into three files: model.py for the architecture, train.py for data loading and optimization, and generate.py for inference. A few hundred lines can expose the essential training machinery, although stronger capabilities still require appropriate data and compute. Perivolaropoulos recalls the concern surrounding GPT-2’s original release as a reminder of how accessible the underlying architecture has become.

For the Colab path:

1. Download the Shakespeare dataset using the companion instructions.
2. Install the dependencies.
3. Assemble and connect the model, training, and generation code.
4. Select a GPU runtime and invoke training with the dataset.

Perivolaropoulos reports about 15 minutes on Google Colab before obtaining good results. He specifically instructs participants to select a T4 GPU runtime; available runtimes and allocation depend on Colab access. The demonstration is a training-and-generation workflow, not a production-serving deployment.

If the default model is too demanding, start with two layers, two heads, and a reduced embedding width, then grow from a working baseline. Compare a 256-token context with 512 and plot training and validation losses with matplotlib.pyplot.

These symptoms direct investigation rather than uniquely identifying a cause. In particular, a plateau does not by itself prove that the dataset is exhausted.

The workshop challenge is to train a model during the session and submit its best verse. Another poetry dataset is allowed; asking ChatGPT to write the submission is not. Participants can regenerate as often as they like, but must retain the checkpoint and generation settings that reproduce the selected output. A QR submission route and possible bracket vote let the room judge creative, funny, or well-made verses, with tentative ElevenLabs swag or credits as prizes.

The reproducibility record consists of the checkpoint, prompt, temperature, and seed used by generate.py, together with any other sampling settings such as top-k. That record connects the displayed poem to the model that produced it. More ambitious experiments include a larger model—Perivolaropoulos considers an 85-million-parameter option probably too large for the exercise—a corpus-trained BPE tokenizer, a longer context, dropout, different learning rates, and earlier stopping.

The closing questions move from verse generation to reasoning. Perivolaropoulos describes a shared architectural foundation: start with a capable base or instruction model, then post-train it with appropriate objectives and high-quality reasoning data. His emphasis is on curated examples, including expert-written solutions, rather than arbitrary internet text. The useful mechanism is that intermediate reasoning becomes additional context. Later answer tokens can attend to a problem restatement, a calculation, or a conclusion already written into the sequence.

Qwen3 is his example of post-training on a common model foundation. Curated reasoning examples are part of that story, but not the whole training recipe: Qwen’s published account includes long-chain-of-thought cold start, reasoning reinforcement learning, thinking-mode fusion, and general reinforcement learning. Perivolaropoulos also connects rapid capability gains to task-focused data, sometimes resembling benchmark tasks; that explanation does not isolate the cause of any particular proprietary model’s improvement.

Long reasoning traces increase the context burden, making attention efficiency relevant again. Perivolaropoulos argues that a GPT-2-style architecture could learn reasoning behavior with sufficient capacity and suitable training, while the tiny workshop model may gain little from it. He cites small Llama models as examples of adaptation without architectural replacement, but does not identify a specific experiment. The actionable distinction is between changing what a model learns and changing the machinery that makes that learning computationally feasible.

Where does high-quality post-training data come from? Perivolaropoulos describes specialist contractors recruited through companies such as Scale AI or through internal labeling teams. Physicists are his example: domain experts produce material that general-purpose annotators may not be equipped to write. His aside about Meta buying Scale is better described by Scale’s announcement: Meta took a minority equity stake, and Scale remained independent.

Producing labels is only the first step. Small defects in valuable training data can affect model quality, so the workflow also needs review. Perivolaropoulos describes experienced labelers moving into QA roles and checking other contributors’ outputs. Organizing this human production-and-review process is part of his own work; an LLM judge is not presented as a substitute for those domain reviewers.

The discussion then returns to seeds. Sampling consumes pseudorandom numbers; setting a seed fixes the generator’s starting state. It does not make every GPU operation deterministic. PyTorch’s reproducibility guidance distinguishes controlling random generators from controlling nondeterministic operations, and warns that identical seeds do not guarantee identical results across releases, platforms, or CPU/GPU execution. For the competition, keep the checkpoint, prompt, sampling settings, software, and hardware environment fixed as far as practical.

Greedy decoding removes the random token draw, although execution details can still affect results. Perivolaropoulos recommends retaining a seed even for nominally greedy generation and mentions an unspecified vLLM behavior. Current vLLM sampling documentation explicitly defines temperature zero as greedy sampling; its handling of small positive temperatures is separate. A seed and greedy selection solve different parts of the repeatability problem.

Moving from text to audio preserves many transformer ideas but complicates representation. Text can teach language structure within an audio-model stack. Audio representation must additionally decide what to preserve: speech, speaker characteristics, music, instrument sounds, or some combination. Generated audio tokens also need a decoding process that turns them into waveforms.

An audience member points out that pitch and sound are less cleanly discrete than written characters. That leads to a discussion of objectives:

• Cross-entropy fits discrete prediction targets, including audio-token systems that use them.
• L2 loss can compare predicted and target mel-spectrogram representations, the TTS example Perivolaropoulos gives.
• KL divergence can train a smaller student to match a larger teacher’s output distribution during distillation.

The representation and supervision determine the loss. These objectives are not mutually exclusive: distillation or post-training can combine losses, and cross-entropy does not become invalid merely because the task involves audio or post-training.

A multimodal model need not build one giant dictionary containing words, sounds, and pixels. The text embedding lookup is one way to obtain vectors; another network can produce vectors too. Perivolaropoulos sketches a video pipeline using a 30-second clip and one frame per second as an illustrative sampling choice:

1. Select frames from the video.
2. Pass those frames through a video encoder.
3. Take its hidden representations.
4. Supply appropriately dimensioned representations to the language model alongside the prompt.

This describes an architectural pattern, not a video implementation added to the workshop model.

In his sequence-level explanation, a video placeholder marks where encoder output replaces ordinary token embeddings. Audio can enter through an analogous encoder path. The receiving transformer requires compatible vector dimensions; its attention layers can then process these representations alongside text. Compatible shape does not establish compatible meaning. When asked whether a horse video’s representation resembles the embedding for the word horse, Perivolaropoulos leaves the question open: the relationship depends on how the system was trained.

Music raises another modeling question: do harmonics and interdependent musical structure favor generating many elements jointly? Perivolaropoulos describes both autoregressive transformers and diffusion models as viable. In an autoregressive system, the representation must make next-token prediction useful for musical structure. He considers diffusion components easier to make work in some audio, music, and image-generation settings, but presents no controlled comparison between the approaches.

The final technical question returns to tokenization: how can audio tokens accommodate different voices, and how long should a token be? The answer is a learned representation, not a person manually assigning each spoken word a code. Perivolaropoulos describes learning recurring patterns from audio data, potentially after converting waveforms into mel-spectrogram arrays. The training distribution matters: a tokenizer learned from music can preserve different structure from one learned from human speech. Covering both well makes the design harder.

The session ends by returning to the small model participants can actually train. Continue the local or Colab run, inspect its samples, and ask for help where the pieces do not connect. The verse competition remains conditional on enough submissions, with a cutoff set to 5:45 and Perivolaropoulos available to troubleshoot. No completed vote or winning model is shown.