Transformers and Attention

Token identities and contextual representations

A forward pass applies the model’s operations once: token IDs become vectors, transformer blocks update them, and an output head converts the resulting representations into task scores. A contextual representation is a vector influenced by other visible positions. Text generation adds a surrounding selection-and-repeat process. [3][4]

A token occurrence is a vocabulary ID at a sequence position; Tokenization explains how text becomes these IDs. An embedding table stores one learned vector—a list of numerical coordinates—per vocabulary entry. The ID selects a row. Its magnitude does not measure meaning. [5]

Repeated IDs select the same initial row. Position and surrounding tokens can then produce different hidden states: the intermediate vectors carried through the model. These coordinates need not have individually named human meanings. Broader questions about representation geometry belong in Embeddings and Representation Learning. [6][7]

Parameters are fitted numerical values; activations are representations computed for the current input. Ordinary inference changes activations without updating parameters. Changing a prompt can therefore change behavior without teaching the checkpoint new weights. Training, fitted state, and inference develops this distinction. [8]

Queries, keys, values and information mixing

Attention forms an input-dependent weighted mixture. A destination’s query scores candidate sources; each source’s key supplies matching features, and its value supplies contributed information. Several sources can contribute simultaneously. The output combines numerical vectors, not token IDs or copied text fragments. These roles are formalized in attention pooling. [9]

$q_i\cdot k_j=\sum_r q_{ir}k_{jr}$ sums coordinatewise products: destination i scores source j. Dividing by $\sqrt{d_k}$ moderates score growth with key width. [10]

Softmax exponentiates scores and divides by their row sum, producing nonnegative weights totaling one. Subtracting the row maximum before exponentiation preserves the result while reducing overflow risk. Normalization couples the source scores; attention cannot generally be rearranged into a simple product that bypasses that operation. [12]

For a constructed example, scaled scores [0, 0] give weights [0.5, 0.5]. Values [2, 0] and [0, 4] therefore produce [1, 2]. The result is a mixed value vector, not a vocabulary distribution. [1]

Visibility and attention masks

An attention mask restricts permitted destination–source pairs. A causal mask permits the current input position and earlier positions, excluding later ones. Including the diagonal is correct: the state at position i predicts the token at i+1. Bidirectional attention permits information from both directions. [13][14]

Padding inserts artificial entries to batch unequal sequence lengths. Excluding padding sources and excluding future sources solve different problems. Teacher forcing, used during training, supplies recorded predecessors rather than generated choices; padding is not one of those meaningful predecessors. [15]

In the SDPA convention, Boolean True means allowed. Forbidden scores become negative infinity before softmax. Every illustrated query has an allowed source. Excluding the example’s second source changes its output to [2, 0]. [1]

A normalized row can still leak future information if the wrong entries remain visible. Removing a future token’s loss does not remove it from attention. Causality here describes prediction-time access; it establishes neither factual correctness nor a causal effect in the statistical sense. [16][18]

Order and positional information

Unrestricted self-attention without positional signals is permutation-equivariant: rearranging input rows rearranges the corresponding output rows. If [A, B, C] becomes [C, A, B], the outputs follow that rearrangement. This differs from invariance, where the output stays unchanged. A fixed causal mask breaks this unrestricted symmetry by changing which sources each row can access. [19]

A relative offset is the difference between two position indices. T5’s bias assigns some offsets the same bucket, deliberately sharing their positional contribution. RoPE changes vector directions instead. Neither mechanism should be confused with deciding whether a source is visible. [22][2]

Multiple heads and residual blocks

Each attention head learns separate projections. H mixtures $N\times d_v$ concatenate to $N\times(Hd_v)$; a learned $(Hd_v)\times d$ output projection restores model width. [10]

Multiple heads permit different mixtures, but do not guarantee indispensable specialists. Ablation studies removed many heads with little change on their tested tasks, while some encoder–decoder layers relied more strongly on multiple heads. Removing heads after training also differs from training a smaller-head model. [23]

The residual connection provides an identity path, letting the branch learn a correction rather than the complete mapping. This supports optimization; it does not guarantee later recovery of every input detail. Both paths still participate in computation. [24]

The feed-forward network, also called a multilayer perceptron or MLP, commonly expands width and projects back: $d\rightarrow d_{\mathrm{ff}}\rightarrow d$. Affine transformations—weighted combinations plus biases—surround a nonlinearity. Parameters are shared across positions. Attention mixes positions; this network transforms features within each position. [20]

Pre-normalization normalizes branch inputs; post-normalization normalizes after residual addition. Neither arrangement defines every transformer. Stacked blocks repeat the operation pattern with their own learned parameters, progressively revising the sequence’s representations. [25][27]

Encoders, decoders and cross-attention

An encoder builds representations of a supplied input. A causal decoder builds representations suitable for predicting a continuation. Conditioning means making predictions depend on supplied information. A decoder-only language model consumes its prompt directly; it does not require a separate source encoder. [28]

Cross-attention uses destination queries and separate-source keys/values. L destinations and S sources yield $L\times S$ weights and $L\times d_v$ outputs; two targets can attend to five sources. [31][1]

The input need not be text. A vision transformer projects image patches into token vectors and attaches a classification head. Its transformer produces representations; the input representation and output head determine the task. Architectural decoding is likewise separate from a tokenizer’s conversion of IDs back into text. [32][28]

Vocabulary readout and autoregressive generation

A language-model head projects a final contextual vector into one logit, or unnormalized score, per vocabulary entry. Final normalization precedes this projection in the illustrated GPT-style architecture. The last input position supplies the distribution for continuing the prefix; the hidden vector itself is not a selected token. [33]

Autoregressive generation predicts from the available prefix, selects a token, appends its ID, and predicts again. Later predictions depend on earlier selections. The model’s parameters remain fixed; the sequence and its computed representations change. One forward pass and a complete response are therefore different units of work. [4][8]

Greedy selection chooses a highest-scoring token; sampling permits alternatives according to a selection distribution. The choice changes subsequent prefixes. LLM Inference covers sampling controls; Decoding and text preservation explains the separate task of reconstructing text from IDs. [35]

Shifted targets and parallel sequence training

A training target states what should be predicted; loss measures prediction error, as explained in Supervision, baselines, and loss. For next-token training, each position’s target is the following recorded token. Cross-entropy penalizes assigning that target insufficient probability. [3][37]

Teacher forcing uses recorded predecessors as decoder inputs. The prediction for C receives recorded B even if the model’s prediction for B was wrong. Generation instead feeds back its own selections. The causal mask, rather than the data loader alone, prevents a training prediction from accessing its answer. [15][14]

Known inputs let positions be processed together within a layer; successive layers still depend on preceding layers. Generation cannot ordinarily select all future inputs together because those inputs depend on earlier choices. Parallel position evaluation is compatible with causal visibility. [20]

Connectivity, computational work and storage

A conceptual N × N score matrix need not be stored in full. FlashAttention preserves dense attention while reducing additional storage, retaining quadratic pairwise arithmetic. Floating-point results can differ slightly. Avoiding materialization changes execution; removing source connections changes the attention operation. [40]

Local attention limits nearby mixing; periodic global layers reconnect distant regions. ModernBERT illustrates this combination. Such a pattern can reduce local-layer work while preserving later communication, but a possible multi-layer path does not establish that the model will use it successfully. [44]

Tokenwise projections and MLPs process each position, with work also depending on feature widths. Dense pair count captures only one component. In vision, doubling both image dimensions at fixed patch size yields four times as many patches and sixteen times as many pairs, without implying sixteen times the total latency. [20][45]

Context access and demonstrated behavior

A controlled evidence-position test moves the same answer-bearing passage while keeping the desired answer fixed. Vary distractors and length separately. Historical studies found position-sensitive performance in several tested models; this motivates measurement, not a universal beginning-or-end placement rule. Failure analysis and competing explanations covers how to distinguish possible causes. [51]

Simple retrieval can succeed while aggregation or multi-step tracing deteriorates. RULER explicitly separates these task demands. A successful needle-in-a-haystack test—finding one requested item among distractors—therefore supports a narrower claim than reliable reasoning across the entire accepted context. [50]

An attention map shows one set of mixing coefficients, not the complete explanation of an answer. Value vectors determine what those coefficients combine. Different weight patterns can sometimes produce the same mixture because their differences cancel against the values. Other heads, residual paths and later transformations further separate one map from final behavior. [53]

A high next-token probability does not establish factual support. If an incorrect token is selected, it joins the prefix and can influence later predictions. Fluent continuation and supported answers require different evidence. [54]

Open questions

1. 01

   Separating positional design from training exposure remains difficult when extending usable context. Both affect long-input behavior, while task difficulty can hide regressions. Progress requires matched training conditions and tests spanning short inputs, distant evidence, distractors and multi-step use—not acceptance of longer sequences alone. [50][49]

2. 02

   Determining necessary head diversity remains task-dependent. Redundant heads and non-unique attention mixtures complicate explanations of individual roles. Progress would identify interventions whose effects reproduce across prompts and tasks while distinguishing post-training pruning from training fewer heads initially. [23][53]

3. 03

   Choosing restricted connectivity that preserves distant reasoning remains an architectural tradeoff. Shifted windows create indirect paths, but path existence does not establish useful information transfer. Progress would demonstrate matched-budget accuracy across local tasks and tasks requiring several distant pieces of evidence. [41]