Low Level Technicals of LLMs

Two implementations load the same Gemma weights. One uses exact GELU; another uses its tanh approximation. Which is correct? The word exact cannot answer that question: the checkpoint expects the computation used during training. Daniel Han, cofounder of Unsloth, opens with this discrepancy, documented in Fixing All Gemma Bugs. His workshop develops the skills needed to investigate such disagreements rather than treating a model’s name as a complete specification.

Other releases provide equally revealing details. Grok includes 30 * tanh(x / 30), which smoothly limits magnitude. Han admits having initially confused multiplication and division while reading the code. Nemotron 340B uses squared ReLU instead of the gated activation pattern familiar from other models. Recognizing the common architecture helps, but its exceptions are often precisely what an implementation must preserve.

The discrepancies can begin before any neural-network arithmetic. Mistral, Llama, and Mixtral tokenizer variants can divide apparently identical text differently, including its spaces. Different token sequences select different model inputs. Han describes differences involving fast-tokenizer variants without resolving every case. Tokenization is a separate debugging problem, not a minor preprocessing detail.

A brief mathematical detour connects familiar PCA visualizations to singular value decomposition, Han’s recommended foundation for understanding matrix computations. He also names QR, LU, Cholesky, and randomized SVD without deriving them. An audience question then makes the implementation problem concrete: extracting Nemotron weights through NVIDIA’s NeMo framework does not supply the modeling code needed for a Hugging Face Transformers port using safetensors. A related Llama 3 70B release is proposed as a possible clue, but Han offers no completed conversion or confirmed architectural mapping. The operations and tensor arrangements still have to be reproduced.

A discussion of LoRA Learns Less and Forgets Less introduces another recurring concern: experimental conclusions depend on the actual training configuration. Learning new information and retaining existing knowledge are different outcomes. Han questions which linear layers received LoRA adapters and how their hyperparameters were configured. His criticism directs attention to what the experiment tested; it does not establish that LoRA universally can or cannot learn new knowledge.

Unsloth’s stated aim is to reduce fine-tuning time and memory use while preserving the intended computation. Han reports roughly 2× faster fine-tuning and 80% lower memory use for supported models. Preserving accuracy without introducing approximations is the optimization goal. These are workshop reports rather than a fully specified benchmark recipe; Han identifies the comparison baseline shortly afterward. Triton supplies specialized GPU kernels while retaining a Python-facing workflow, making smaller GPUs useful for fine-tuning.

His Tesla T4 versus P100 discussion illustrates why hardware age and purchase price are insufficient guides. He cites 65 TFLOPS for one T4 and 130 for two as arithmetic rates, not measured training throughput. Kaggle’s cited 30-hour weekly allowance and Colab’s free or paid access describe the workshop’s historical options. The enduring comparison is between the arithmetic paths a workload can actually use. Before comparing advertised FLOPS or rental prices, identify the numeric format, dense versus sparse arithmetic, and whether the model’s kernels use that acceleration.

Sparsity makes the distinction especially clear. The advertised 2× sparse arithmetic rate assumes a supported sparsity pattern and compressed matrix representation. Hardware can avoid work associated with zeros, but erasing half the weights from an existing dense model changes its function. Recovering useful behavior may require additional training. Han mentions PyTorch and xFormers support in this context, then considers sparsity and FP8 as possible explanations for other companies’ speed claims. He explicitly treats those explanations as guesses.

Han identifies Hugging Face with FlashAttention-2 as the baseline for Unsloth’s comparisons and notes that the displayed benchmark predates newer memory optimizations. The user-facing entry point conceals the kernel and memory-management work underneath:

from unsloth import FastLanguageModel

Understanding that work requires separating arithmetic throughput, end-to-end execution time, and memory consumption. The import exposes an interface, not a complete fine-tuning procedure or benchmark configuration.

Asked whether linear or cosine learning-rate schedules offer a generally superior recipe, Han recommends small experiments. Schedule shape is one choice alongside learning-rate magnitude, dataset composition, model size, and tied versus untied weights. Token accounting also matters: fifteen trillion token exposures could mean fifteen trillion distinct tokens or, hypothetically, five trillion tokens processed three times. He does not establish which interpretation applies to a particular release. His suggestions about one pass for pretraining or approximately three epochs in other training settings are starting heuristics, not universal optima.

An audience member describes a Gemma 2B fine-tune trained before the implementation fixes, using a searched learning rate and eight GPUs. It was reportedly usable before the fixes and improved on a leaderboard afterward without retraining. The exchange establishes what stayed fixed: the checkpoint. What changed was the code executing it. Han declines to identify a cause because several fixes had been applied together. Without a controlled comparison, the anecdote cannot isolate the effect of GELU, the learning rate, or any other individual change.

The next optimization combines gradient checkpointing with offloading saved activations to system RAM. Checkpointing reduces retained intermediate results by recomputing them during the backward pass; offloading changes where selected saved data resides. For Unsloth’s nonblocking checkpoint-offload implementation, Han reports roughly 4× context capacity with only 1–2% more execution time. The report concerns that transfer arrangement, not arbitrary CPU offloading, and does not specify the hardware or sequence-length configuration. Movement must overlap useful computation, and an operation still cannot use a tensor before its transfer finishes.

Han distinguishes system RAM from disk and argues for using memory before resorting to much slower storage. Nonblocking calls help hide transfers; they do not make movement free. The optimization is therefore a scheduling problem as well as a storage problem: decide what to retain, what to recompute, and when transferred data must be available again.

Transformers model sequences beyond language; Han mentions Sora while establishing that broader scope. For the language-model walkthrough, he starts with the decoder pattern associated with GPT-2. He uses Transformer++ as a descriptive grouping for subsequent choices: rotary position embeddings, SwiGLU, RMSNorm, and omitted biases. His worked example is Llama 2, whose architecture he compresses into a slide of equations.

The central block repeats a sequence of operations. Normalize a branch, apply attention, and add its result to a residual path. Normalize again, apply the multilayer perceptron, or MLP, and add another residual. Repeat the block L times, then use a final normalization and output projection to obtain token scores. The equations map the forward computation; they are not an extra mathematical description disconnected from the code.

An audience member proposes a Plinko analogy: a prompt passes through successive layers toward an output. Han offers fashion designers instead. Each designer changes the outfit received from the previous one. The useful part of this image is successive transformation: each layer modifies a representation rather than simply routing an unchanged object through a maze.

Matrix notation makes those transformations inspectable. Capital letters generally denote matrices; lowercase weights may denote vectors, and indexed summations indicate operations over rows or columns. Gathering token vectors into matrices exposes work that hardware can perform together. For each expression, identify the input shape, the operation, and the resulting shape.

Han displays four sentences, beginning with Hello, my name is Daniel. The audience looks for semantic relationships: greetings, similar verbs, even word2vec arithmetic. His intended first operation is simpler. Ignore spaces and keep punctuation attached to the neighboring word. The sentences contain five, six, eight, and seven components under that convention. The first becomes Hello,, my, name, is, and Daniel. Assign integer IDs—Hello, becomes 0, my becomes 1, and name becomes 2—and the workshop has a tokenizer.

The audience quickly finds several different limitations:

• Attaching punctuation makes Michael and Michael! separate vocabulary entries.
• Whole-word entries make the vocabulary grow whenever a new word appears.
• Stemming can group skip, skipped, and skipping, but removes distinctions between their forms.
• Lowercasing reduces separate entries while also removing information carried by capitalization.

These are choices about what information the encoding preserves, not merely ways to make a dictionary smaller.

Uncased BERT illustrates that preprocessing belongs to a particular model and task. A transformation suitable for one application need not preserve what a generative model should reproduce. The discussion then reaches BPE and WordPiece: subword approaches that construct reusable pieces from smaller units using corpus statistics. Their training rules are not identical, and the workshop does not derive either algorithm. Han keeps the simple five-token convention for the remaining examples so the arithmetic remains easy to follow.

With tokenization fixed, the first task is to predict my from Hello,. Later tokens must remain unavailable to the computation making that prediction. Otherwise training exposes information the model will not have while generating.

Audience questions separate the stated method from its implementation. A paper can describe a sensible information boundary while code violates it; a methodology can also be flawed before anyone implements it. Han urges scrutiny of unexpectedly strong results and assigns research leadership responsibility for checking them. A performance jump is a reason to trace the available information, not proof that a particular kind of leakage occurred.

Another question concerns train/test splits. Examples or related material can cross an evaluation boundary even when attention inside each sequence is correctly masked. These are separate problems: causal visibility controls what a prediction can inspect, while dataset construction controls what evaluation measures. Inspect the examples and split design; use shuffling and stratification where appropriate rather than assuming one split recipe fits every task. Returning to the sentence, once my has been generated, the prefix becomes Hello, my, and the next target is name. Each generated successor joins the prefix for the next step. Training must reproduce that information structure without supplying the answer in advance.

The next representation is a vector of numbers for each token. Every vector has the same width. Han invents three-coordinate examples, prompting an audience member to challenge his suggestion of infinitely many possible representations: actual floating-point precision is finite. The infinity argument only applies under an infinite-precision assumption; it does not describe a computer’s storage or a model’s useful capacity.

The invented values also contain a deliberate trap: coordinates such as 102 and 123 sit beside much smaller numbers. Initialization scale matters because those values enter repeated transformations. Random initialization does not mean arbitrary magnitude. Arranged as a table, the token representations become trainable rows with a common number of columns. The ellipsis in the slide represents additional coordinates chosen by the model designer, not missing values. Randomly initialized rows will generally differ; there is no separate requirement to enforce uniqueness.

The text itself supplies supervision. Shift the sequence one position so each input aligns with its successor. In this example, EOS marks the end of the sequence. This Python listing expresses only that alignment:

tokens = ["Hello,", "my", "name", "is", "Daniel."]
targets = tokens[1:] + ["EOS"]

for token, target in zip(tokens, targets):
    print(f"{token} -> {target}")

The pair name → is does not mean the model sees only name. Its representation must incorporate the permitted prefix. Target alignment and visible context are different parts of the training setup; attention supplies contextual mixing, while the MLP transforms features at each position.

An audience question extends this to multiple future tokens. Additional objectives can align a representation with offsets of two, three, or more positions. That changes the output arrangement and training objective. It does not automatically translate the number of targets into an equal generation speedup: inference still needs a way to use those predictions. Han expresses a preference for the simpler one-successor objective while acknowledging the alternative’s possible inference benefits.

The next questions clarify the distinction between an ID and a vector. A vocabulary with 32,000 entries uses IDs from 0 through 31999. Each ID selects a row in the embedding matrix; it does not specify how many coordinates the row contains. Vocabulary size counts rows. Embedding width counts columns. The slide’s five labeled token rows and numeric matrix illustrate this lookup; their displayed coordinates are teaching values, not measured learned embeddings.

Widths such as 2,048, 4,096, or 6,144 provide different amounts of trainable freedom and require different amounts of computation. The audience challenges the idea that increasing width must always improve quality, and Han agrees that the choice needs experiments. A dense learned vector and a one-hot vocabulary indicator are also different constructions; counting their dimensions alone does not explain their behavior.

Unused vocabulary entries create a practical complication. Entries may be reserved for later use, and matrix sizes may be padded for execution efficiency. Neither guarantees that every associated vector has been meaningfully trained. Han describes NaNs when certain unused embeddings in the Llama 3 base model enter fine-tuning, and an Unsloth repair that initializes detected problematic entries from the mean embedding. His tentative explanation involving a particular padding multiple is not a general hardware rule.

An audience member also notices that the example contains EOS but no beginning marker. Han acknowledges omitting BOS for the demonstration. In an actual pipeline, beginning and ending markers must follow the model’s expected convention; a teaching shortcut is not a replacement for that preprocessing contract.

Why not train by separately running Hello,, then Hello, my, then every longer prefix? That repeats computation. For fixed model dimensions, independently recomputing dense attention for every prefix has O(N³) total attention work, whereas one causally masked sequence pass has O(N²) attention work. The cubic comparison sums a quadratic attention cost over successively longer prefixes; it is not a claim that every component of transformer training has the same scaling.

A causal mask allows each position to use its current token and earlier context while blocking later positions. All rows can therefore be computed together, with different visibility in each row. In an additive mask, allowed entries receive zero and blocked entries receive negative infinity before softmax. The static example below keeps the sentence fixed and places each row’s known training target beside its permitted context. The target is supervision, not an input shortcut.

Han prefers matrix operations to metaphors about queries searching for keys. Starting from token representations X, learned projection matrices produce Q = XW_Q, K = XW_K, and V = XW_V. The projection matrices are parameters; Q, K, and V are computed activations. Their shapes reveal what the next operation can do.

For one head with N token positions and query/key width d, multiplying Q by Kᵀ produces an N × N score matrix. Each row compares one query position with all key positions. Scale the scores before applying row-wise softmax. Subtracting the row maximum before exponentiation avoids unnecessarily large positive exponential arguments without changing the mathematical softmax distribution.

Softmax supplies nonnegative weights whose rows sum to one. Its nonlinear normalization also prevents simply reassociating attention into an equivalent Q(KᵀV) product. Replacing softmax is an architectural change, not a harmless rearrangement of multiplication. These weights ordinarily mix value vectors; attention does not need to sample one value from each row.

Combining these operations with the causal mask M introduced earlier gives:

Multiplication by V returns one output row per query position. Each row is a weighted mixture of the value vectors that position may use. The square score matrix is an intermediate; the result returns to a token-by-feature representation.

The root-mean-square operation in the Llama discussion is RMSNorm. For a token row x, learned coordinate scales w, and stabilizing constant ε, its essential expression is:

Unlike mean-centering LayerNorm, this operation does not first subtract the row mean. Han emphasizes its practical role: controlling the scale passed into subsequent computation so training is easier to optimize. The learned scale still allows individual coordinates to change.

The Triton forward kernel contains relatively little arithmetic: load values, calculate the reduction, normalize, multiply by the learned scale, and store. Much of its surrounding code handles addresses, masks, and data movement. Those lines are separate from the formula but remain essential to computing it correctly.

The backward pass must propagate gradients through the reduction and accumulate gradients for the learned scale. Han points to the Triton layer-normalization tutorial for a fuller kernel treatment; that tutorial’s LayerNorm should be distinguished from the RMSNorm formula above. He defers a general backpropagation tutorial. The immediate lesson is that a compact forward expression does not eliminate the work of deriving and implementing its reverse computation.

The question of what rotary position embeddings do initially sends the workshop into context extension. An audience member asks about moving from a short training context to a much longer one, mentioning YaRN. Han presses a distinction that survives the tentative details of the exchange: a large corpus is not the same thing as a long example. FineWeb’s cited fifteen trillion tokens across a dataset do not automatically provide supervision for dependencies spanning a ten-million-token context.

The discussion moves through better datasets, synthetic examples, and material derived from a large codebase. Claude is proposed as a possible data-generation aid, not a demonstrated solution to the long-range training problem. Representing additional positions, executing attention over them, and using their information reliably are separate capabilities. Han questions what behavior supports very large context claims; ring attention enters the conversation as another part of the execution discussion, without a technical walkthrough.

Returning to the original question, word order matters: the same token in different positions should participate differently in attention. RoPE supplies position-dependent rotations using sine and cosine factors. An audience question recalls the uncased BERT example, but capitalization and token position carry different information; encoding one does not recover the other. The ensuing conversation names the RoFormer paper and contrasts rotary encoding with earlier positional approaches. Han declines an expert comparison of YaRN variants while the audience raises the behavior of short prompts under scaling intended for long contexts.

A follow-up separates positional scaling from execution cost. Changing rotary scales modifies how positions beyond the original range are represented; it does not by itself establish useful extrapolation or supply a complete YaRN recipe. A million-token dense attention matrix has a trillion token-pair positions. Linear transformers are mentioned as another research direction, but the workshop does not develop them into a replacement for the attention mechanism already explained.

Another question asks when to use in-context examples rather than fine-tuning. Examples in a prompt change the information supplied to the forward pass; fine-tuning uses gradients to change parameters. Han’s practical order is to try a few-shot prompt first, then investigate fine-tuning if the behavior remains inadequate or repeatedly supplying examples is inefficient. He mentions recent work relating in-context learning to trees, without identifying the paper, and recommends experimentation rather than a universal ranking of the methods.

Back in the code, RoPE combines a cosine-scaled representation with a sine-scaled rotated copy. In the half-split convention Han shows, rotate_half maps [x₁, x₂] to [-x₂, x₁]. The slices and concatenation implement a structured linear transformation rather than an arbitrary rearrangement.

For one coordinate pair, let c and s be the positional cosine and sine. The forward operation and its backward propagation are:

Here g₁ and g₂ are the incoming gradients with respect to the outputs. Applying the transpose of the rotation produces the sign change Han highlights. This accounts for the gradient arriving from later computation, rather than saying that the derivative is simply a transpose.

Symbolic tools can help derive the local formula, but the implementation must still pair the correct coordinates and broadcast factors over the correct tensor dimensions. Algebra does not verify memory addressing. Writing the operation in a useful mathematical form is the beginning of that check, not its completion.

Attention mixes information across positions. The MLP transforms features within each position. Han mentions GLU Variants Improve Transformer and contrasts a plain MLP with a gated one:

The additional projection and element-wise product distinguish the gated form. SwiGLU uses SiLU for its gate activation.

Nemotron’s squared-ReLU choice returns as a reason not to treat gated MLPs as mandatory. Han then recommends Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. The relevant comparison concerns knowledge capacity in controlled experiments, particularly small models: Han highlights settings in which a plain MLP fares better than a gated one. That finding should not become a universal architecture ranking across tasks, training regimes, and scales. His broader request is for ablations of activations, biases, and other components rather than inherited assumptions about which choices must help.

The forward kernel again has a small arithmetic core inside data-loading machinery. Han recommends considering torch.compile before writing manual Triton kernels. A following question clarifies the projection names: gate_proj, up_proj, and down_proj are learned transformations. Up and down refer to expanding and reducing feature dimensions, not directions in a diagram.

Backward kernels must account for the activation derivative and both sides of the element-wise gate. Asked how he verifies them, Han first mentions plotting formulas in Desmos. The audience pushes further: what if data loading is wrong? His answer is to isolate and test every component. Matching scalar curves cannot compensate for reading the wrong tensor elements or writing results to the wrong locations.

The output layer produces vocabulary logits, and cross-entropy compares them with the target token. Reductions complicate both the forward calculation and its derivatives. Han suggests expressing sums as matrix operations where that makes differentiation easier to reason about: a row sum can be written as multiplication by a vector of ones. He then returns to maximum subtraction for stable exponentials.

For logits z and target index y, the per-position loss and stable normalizer can be written as:

The maximum is restored outside the logarithm, preserving the result mathematically. This is different from clipping large logits or changing their relative probabilities.

Han’s chunked cross-entropy implementation divides a large vocabulary row into smaller pieces. If each chunk produces its own log-sum-exp value a_c, another log-sum-exp combines them:

The chunks contribute to one global normalization. Adding their logarithms directly, or treating each chunk as an independently normalized vocabulary, would change the loss. Work distribution, temporary storage, and backward propagation must all respect that combined reduction.

At the workshop, Han describes chunking as present in Unsloth and discusses possible future compiler support. He cites 65536 as a threshold motivating chunks in the implementation under discussion. That kernel constraint should not be mistaken for a universal CUDA limit on vocabulary size: the underlying issue is how a particular reduction distributes work and fits its resource limits.

The source walkthrough turns the equations into a reading strategy: find a component’s class, locate forward, and identify the arithmetic before following every setup or distributed-execution branch. In modeling_llama.py, LlamaRMSNorm exposes the familiar reduction and rescaling. Rotary embedding code prepares positional factors, including precision handling that exists because a simpler mixed-precision path produced incorrect values. Not every complicated-looking line is expendable.

An audience member asks whether compiler-generated kernels can match custom Triton kernels. In the compiler discussion, Han reports an approximately 30% remaining speed advantage for Unsloth. He corrects a spoken 30× slip, but the exchange does not pin down a complete compiler baseline or timing configuration. This is separate from the roughly 2× comparison against Hugging Face plus FlashAttention-2. Compiler tooling can incorporate kernels from specialized packages, so the practical question is how compilation and custom kernels interact in the complete training path.

The MLP reduces conceptually to down_proj(act_fn(gate_proj(x)) * up_proj(x)). The nearby repeat_kv operation introduces another mechanism: fewer key/value heads can be shared across several query heads. This reduces the distinct key/value state relative to giving every query head its own key/value head. It is separate from caching previous tokens’ projections during autoregressive decoding.

Attention’s implementation reveals projections, reshaping, rotary application, key/value sharing, score multiplication, softmax, value mixing, and output projection. Tensor dimensions are the main bookkeeping challenge. The pretraining_tp branches concern tensor parallelism; FlashAttention-2 and PyTorch’s scaled dot-product attention supply alternative execution paths. Bracketing those branches during a first conceptual reading is different from deleting them from a working implementation.

The decoder combines the components in the expected order: normalization, attention, residual addition, then normalization, MLP, and another residual addition. An audience member notices that the residual was saved before normalization. The attention branch therefore receives a normalized representation while the bypass retains the incoming one.

The structure is y = x + Attention(Norm(x)). Both terms have compatible shapes, and automatic differentiation follows both computational paths. That explains how gradients can be calculated; it does not prove that moving normalization elsewhere leaves the model unchanged or trains equally well. The placement is part of the architecture. Han’s suggestion to add normalization more widely is a hypothesis about stability versus execution cost.

The conversation broadens to TransformerLens and sparse autoencoders as possible tools for understanding such choices, but neither is demonstrated. Han has not used the named tool and keeps his explanation focused on scale. He recalls internal covariate shift from discussions of batch normalization without establishing it as an explanation of RMSNorm. Repeated transformations can amplify values, and normalization controls what enters the next branch. His repeated-multiplication example supplies intuition for growth, not a calculation establishing float32 overflow after 32 doublings.

After the decoder layers, final normalization and lm_head map hidden representations to vocabulary logits. Logits are scores, not yet probabilities. The historical implementation casts the loss-path logits to float32 because exponentials and reductions are sensitive to numeric format. This is a targeted precision choice, not a requirement to convert every operation and parameter to full precision.

A question about the layer loop clarifies what repetition means. for decoder_layer in self.layers visits distinct decoder layers with their own learned parameters. It does not ordinarily generate fresh weights during the forward pass or repeatedly apply one shared set. Increasing the number of these layers adds parameters and computation. The earlier target shift finally appears as slicing: logits[..., :-1, :] aligns with labels[..., 1:] before cross-entropy.

The Llama 2 versus Llama 3 question reinforces the reading method: recognize the shared decoder pattern, then inspect dimensions, vocabulary, positional settings, and other configuration choices. Han emphasizes more training-token exposure and a larger vocabulary. Family resemblance is useful, but it is not a substitute for reading the actual model configuration or evidence that every architectural detail is identical.

Asked why he favors Triton over CUDA, Han emphasizes implementation effort and dependence on a specific hardware stack. Direct CUDA offers hardware-specific control; a compiler can handle more backend work. The complete application still depends on compatible kernels and attention libraries, with scaled dot-product attention suggested as an alternative to xFormers. An audience member reports training with Triton on one AMD MI300X; Han distinguishes that experiment from Unsloth’s lack of official AMD support at the time. One successful machine, compatible dependencies, and a supported training system are different claims.

The Gemma investigation returns to the opening discrepancy with a concrete method. Read the paper, establish the intended computation, run reference and candidate implementations with matching weights and inputs, and collect corresponding layer outputs. BOS handling and a paper discrepancy are among the reported issues. Han’s Gemma comparison measures the L2 difference between corresponding layer outputs and displays it on a logarithmic scale. The graph spans 18 layers and measures implementation agreement, not training loss or task accuracy. Each curve represents a different combination of changes.

Han identifies the RoPE float32 repair as producing a particularly substantial change in the displayed discrepancy. The audience also notices rises and drops across layers. He offers a tentative explanation involving how well different layers were trained, but the comparison does not establish that theory. What the graph directly reveals is where two computations disagree.

Why choose the black curve rather than the apparently attractive blue one? Multiple implementation differences can interact. Han chooses the combination intended to reproduce the original computation rather than treating one appealing local change as sufficient. The reference cannot be selected merely by whichever candidate produces a nicer-looking curve: the checkpoint was trained under particular activation and numerical choices.

The following questions distinguish architecture from learning. Text supplies successor-token labels; backpropagation supplies gradients for updating learned weights, and PyTorch autograd handles those derivatives in an ordinary implementation. For the embedding table and lm_head, Han describes using a learning rate one tenth of the rate applied elsewhere. He reports a benefit but gives no task or score here. This is a parameter-group choice, not a demonstrated universal optimum or a complete adaptive layer-wise optimizer. He also names LARS in the discussion of layerwise optimization.

Clarifying the graph’s axes matters: the desired raw discrepancy approaches zero, while its logarithm does not approach zero. The plotted quantity is log(||h_port - h_reference||₂), with no logarithm base specified. A tokenizer question then raises a more expensive kind of change. Altering segmentation or token-to-ID assignments can break the associations learned by the checkpoint. Han recounts a tokenizer issue discovered late in TinyLlama training and argues for small experiments before scaling. Major tokenizer replacement can require extensive retraining, but the adaptation required depends on what changed; not every repair requires starting over.

Opening modeling_gemma, Han points to normalization differences that copying a Llama implementation can miss. The important question is when values are upcast, when scaling happens, and when results return to a lower-precision dtype. For the Gemma operation under discussion, intermediate calculation remains in float32 until the appropriate final cast. Moving that cast changes rounding, even when the algebra written on paper looks unchanged.

The hardware motivation leads to floating-point layouts. Exponent bits primarily determine range; fraction bits determine precision within that range. The formats discussed allocate their bits differently:

Thus bfloat16 retains a broad exponent range while representing values more coarsely than float32. Equal storage width does not imply equal numerical behavior.

Han sketches arithmetic cost using exponent bits + fraction bits², yielding 537 for float32, 57 for bfloat16, and 105 for float16. These illustrate his circuit-cost intuition, not measured transistor counts or guaranteed speed ratios. T4 Tensor Cores motivate the practical interest in supported low-precision arithmetic. The discussion continues through FP8 layouts and smaller formats, where data movement, energy, and supporting circuitry limit the benefit of shrinking the multiplier. An audience member challenges the sketch using GPU specifications with equal BF16 and FP16 tensor throughput. Hardware provisions particular execution units; bit allocation alone does not determine their throughput. TF32 likewise distinguishes reduced arithmetic precision from the storage and accumulation choices around an operation.

The low-bit discussion reaches Blackwell FP4 support, then BitNet and 1.58-bit weights. Han focuses on additional training machinery, including straight-through estimation, and expresses a preference for hardware-supported floating-point formats. That preference is not a measured equivalence between BitNet and FP4. Weights, activations, and gradients can use different representations, and his prediction of diminishing returns below FP4 remains a forecast. An audience member reports Mixtral 8x7B running on a phone at roughly twenty-something tokens per second using two-bit precision and UFS 4.0 storage. The report does not supply a reproducible hardware and runtime configuration. Han also mentions allocating fewer bits to MLPs than attention, reinforcing that storage precision need not be uniform across the model.

Questions about the sign bit and softmax bring the discussion back to representation. A conventional sign bit supports positive and negative values, but alternative encodings need not use the same layout. Softmax is exponential normalization, not linear compression into fewer bits. Subtracting the maximum controls intermediate magnitudes while preserving relative probabilities: the common exponential factor cancels from numerator and denominator. It does not flatten a distribution that is mathematically concentrated on one entry.

Finally, asked whether Gemma’s normalization upcast is slower or inherently more accurate than Llama’s arrangement, Han does not supply a measurement. His reason for preserving it is fidelity: it belongs to the computation used by Gemma. A port is not the place to silently substitute a different numerical experiment.

The practical investigation requires several implementations side by side: DeepMind, Hugging Face, and Keras. Compare their operations line by line, consult the paper, and ask maintainers about discrepancies. Automated tests can expose disagreement and evaluate candidate changes, but they need an established reference. They cannot infer the intended training computation merely by voting among ports.

RoPE precision supplies a concrete failure. In a sequence with 8,192 positions, the intended final indices are 8189, 8190, and 8191. The notebook’s bfloat16 calculation instead repeats 8192, collapsing distinct positions and producing a value outside that sequence’s zero-based index range. The contrasting output retains distinct terminal positions. The repair constructs or recomputes positions at sufficient precision before calculating rotary factors. Upcasting already-rounded values to float32 cannot recover the lost distinctions. Repairing those positions also leaves the previously identified activation mismatch to be checked separately.

The activation investigation returns to exact and approximate GELU. Keras and other implementations had chosen different variants. The PyTorch 2.13 GELU reference, as related API documentation, distinguishes the error-function expression from the tanh approximation. Written as scalar Python functions, the formulas Han plots are:

from math import erf, pi, sqrt, tanh


def gelu_exact(x: float) -> float:
    return 0.5 * x * (1.0 + erf(x / sqrt(2.0)))


def gelu_tanh(x: float) -> float:
    return 0.5 * x * (1.0 + tanh(
        sqrt(2.0 / pi) * (x + 0.044715 * x**3)
    ))

In Desmos, Han overlays the functions and their derivatives. Their visual similarity explains why this mismatch is easy to overlook, but similar curves do not establish identical computation. Nor does a tiny integral of a signed difference prove tiny error everywhere: positive and negative contributions can cancel. The relevant implementation question remains which function the checkpoint expects.

Han again identifies the positional repair as the more consequential fix in this investigation. He then compares precision settings: distinctions visible in float32 can become harder to separate in float16 or bfloat16 because rounding changes the observed differences. Candidate repairs therefore need comparison under the intended numerical regime as well as inspection of their formulas. For the displayed Gemma fine-tuning notebook, Han cites roughly 2× speed and 60% lower memory use, describing the memory figure as already outdated. As with the earlier reports, these figures do not specify a complete hardware and training configuration.

The Phi-3 investigation starts with config.json. Han inspects special-token IDs, padding, layer count, and positional settings before reading more modeling code. The medium checkpoint opened on stage reports 40 layers and a context setting of 131072: 128 × 1024, rather than 128,000. These values describe that configuration, not every model carrying the Phi-3 name.

Moving to another variant, he flags a sliding-window setting of 2047 where he expects 2048, recounting a fix that was subsequently reverted. The general debugging issue is boundary semantics: does a window parameter include the current position, count preceding positions, or specify a total span? An unusual number is a useful lead, but being unlike a power of two is not by itself proof of an error. The configuration and the consuming attention code must agree.

Then the live inspection catches Han’s own configuration problem. While showing Unsloth’s Phi-3 files, he notices an incorrect padding-token setting. His warning about sharing padding and EOS IDs is especially relevant when training labels are masked by token identity: masking every occurrence of the padding ID can also remove EOS supervision if the IDs are shared. The safe behavior depends on the collator and masking logic, not just the spelling of the token.

Han says the loader repairs the setting while the saved configuration still needs attention. This makes an otherwise abstract distinction visible: a runtime correction and the metadata distributed with a model can disagree.

Phi-3’s fused QKV projection raises a different issue for LoRA. Fusing the base projections can preserve the same dense calculation, but attaching one fixed-rank adapter to the combined matrix is not the same parameterization as attaching independent adapters to Q, K, and V. A combined adapter can share an input-side low-rank factor across the outputs; separate adapters allow independent factors. Han describes unfusing the projections in Unsloth’s version for this adaptation setup. Compare adapter targets and ranks before interpreting a quality difference: this is not a general proof that fused projection arithmetic is less accurate.

Han introduces an Ollama export workflow by returning to the problem of running a fine-tuned model somewhere else. Ollama provides an inference runtime for a selected model, but moving weights into it does not automatically preserve the input structure those weights expect. Formatting must survive the handoff too.

An audience member describes difficulty generating Ollama Modelfile definitions automatically. Han shows Unsloth’s chat-template machinery, including model-specific boundary tokens and role formatting. A chat template serializes messages into the sequence the model consumes; the wrong serialization can produce poor responses from otherwise usable weights. He shows Vicuna, Alpaca-style, Gemma, and Llama 3 formats. Related Hugging Face chat-template documentation explains the role markers and special-token handling involved, without defining the historical Unsloth implementation.

The automatic conversion code includes parse_combined_prompt, which Han describes as quadratic in the shown implementation. Its job is to infer enough structure from the supplied format to construct a Modelfile. He identifies the code as Apache-licensed and is candid about its complexity. The difficulty lies in preserving the boundaries and separators around messages, even though those details can look insignificant when rendered as ordinary text.

The notebook begins with an Alpaca-style dataset containing GPT-4-generated responses, then broadens to CSV input. A spreadsheet can have many columns; the demonstrated fine-tuning workflow expects an instruction and a response. Its solution is to serialize the input columns into one prompt while keeping the target separate. In the Titanic example, passenger fields such as siblings and spouses become labeled information in the instruction. Optional template fragments appear when their source fields exist. Han notes that the demonstrated syntax may change. This two-field interface belongs to the workflow, not to an intrinsic limitation of language models.

Custom templates replace the assumption that every dataset already fits the notebook’s earlier Alpaca prompt. Additional inputs can be merged into the instruction, with optional fields included when present. Whether that preserves useful meaning depends on the formatting: values need enough labels and context for their role in the task to remain clear. Flattening columns should not mean throwing their distinctions away.

One historical requirement is unusually specific: the automatic Modelfile construction shown here needs two instruction/response repetitions in the template example. Han explains that a single pair can leave dangling newlines and incorrect boundaries; the implementation rejects an insufficient example. He demonstrates the same requirement with the Llama 3 template. This concerns inferring the repeating format. It does not mean every training conversation or inference request must contain two exchanges.

The notebook checks inference with a Fibonacci continuation and shows how the template accepts multiple messages. Such an example is a useful smoke test of the path through the model, not evidence that fine-tuning created the underlying mathematical capability. Saving then presents a choice: retain the LoRA adapters separately or merge the update into the base weights. In the notebook example, Han describes approximately 100 MB of LoRA adapters versus approximately 16 GB for merged weights. These sizes belong to that example; adapter rank, targeted modules, model size, and storage precision affect the artifacts.

The demonstrated Ollama route proceeds through a merged model and GGUF:

1. Merge the adapters into the base weights for the demonstrated export path.
2. Choose the GGUF quantization method explicitly; Han mentions eight-, four-, and five-bit options.
3. Save the quantized model and generate its Ollama Modelfile from the chat template.

The workflow can produce multiple quantized outputs in one operation. Current Unsloth GGUF export documentation describes the export interface today, distinct from the notebook interface shown on stage. The notebook frame shows quantization choices and conversion logs at this export step.

Han then prints the generated Modelfile and walks through using it in the serving workflow. Inspecting that definition connects the exported weights to the intended prompt format. Conversion logs establish the export activity shown in the notebook; they do not by themselves establish successful inference in a separate runtime.

The CSV notebook repeats the process with the Titanic dataset and a more elaborate column-merging template. Upload the table, serialize its input fields with the supplied template, and follow the same training and export route. Han presents these notebooks as experimental material for workshop participants to try and report problems with. The tabular example reaches the same deployment path as the instruction dataset; it does not require treating each spreadsheet column as a separate model input.

The final substantive question asks whether merged tabular values should become English sentences or JSON. After clarifying that JSON can be placed inside the instruction, Han expresses a preference for prose. The audience asks whether he measured the difference; no comparison is reported. Both formats can carry the same row values into the same task. Holding the instruction and underlying values fixed while changing only their serialization would test the preference instead of assuming it.