Tokenization

Text as vocabulary IDs

A vocabulary is a finite inventory of entries with integer IDs. A token occurrence references one entry at one sequence position. Encoding segments text and produces IDs; tokenizer decoding reconstructs text from IDs. Entries can represent whole words, fragments or characters, rather than one uniform linguistic unit. [1]

Suppose a vocabulary contains cat → 7, space → 8 and dog → 9. The text cat cat becomes [7, 8, 7]: three occurrences referencing two distinct entries in a three-entry inventory. The repeated 7 preserves identity. Its numerical distance from 9 does not measure similarity in meaning. [1][2]

Text units and preprocessing

A Unicode code point is a numbered position in Unicode's codespace. Character encodings represent text using code units: UTF-8 uses bytes, while UTF-16 uses 16-bit units. A programming language's string length may count these units rather than visible characters. [3]

A grapheme cluster approximates a user-perceived character. A base letter followed by a combining accent can form one cluster despite containing two code points. Neither grapheme boundaries nor byte boundaries necessarily coincide with model-token boundaries. [4]

Normalization selects a representation under an equivalence rule. NFD canonically decomposes characters; NFC also composes them where possible. NFKD and NFKC additionally apply compatibility transformations, such as replacing the fi ligature with fi. Compatibility normalization can discard meaningful distinctions. Lowercasing and accent removal are separate transformations, not synonyms for Unicode normalization. [6]

Pre-tokenization partitions text into regions before segmentation. A punctuation-splitting configuration separates the apostrophe in I'm; individual-digit splitting makes 911 three regions. Later merges cannot cross those imposed boundaries. These stages are configurable, so case and whitespace behavior must be checked for the selected tokenizer. [7]

Subwords and unfamiliar-text coverage

A subword is a reusable text segment, not necessarily a root or syllable. An out-of-vocabulary string lacks a direct entry; smaller pieces may still represent it. An unknown token is a placeholder used when the configured representation cannot cover the input. [8]

Whole-word coverage demands a growing inventory; character sequences require more positions for the same passage. Subwords balance these pressures. For a small model, a compact vocabulary can be useful even when it produces longer sequences. Inventory size alone therefore cannot identify the best representation. [10][11]

Complete byte coverage starts with every byte value. Byte fallback instead uses UTF-8 byte tokens when ordinary pieces cannot represent a character. SentencePiece configuration distinguishes fallback, character coverage and normalization. Representing every remaining byte cannot restore text already changed by normalization. [12]

Learning and applying BPE merges

Byte-pair encoding, or BPE, adapts a compression idea: repeatedly replace frequent adjacent symbol pairs with larger symbols. A corpus is the collection used to learn these rules. Pair frequencies include repeated occurrences of each word; merges change the counts available at the next iteration. [13]

# Character-initialized, word-bounded training.
segments = split_words_into_characters(corpus, end_marker)
rules = []
for _ in range(merge_budget):
    counts = weighted_adjacent_pair_counts(segments, frequencies)
    if not counts:
        break
    pair = highest_count_pair(counts, tie_order)
    segments = replace_nonoverlapping_pairs(segments, pair)
    rules.append(pair)

The figure's two rules encode the new word lowest as l|o|w|es|t|◇, then l|o|w|est|◇. The end marker ◇ is structural, not literal source text. [16]

Encoding uses fixed merge ranks: priorities learned beforehand. It does not recount request-local frequencies. Byte-level implementations can start each permitted region with UTF-8 bytes rather than characters; preprocessing still restricts which pairs may merge. [17][18]

# Encode one permitted region with fixed ranks.
pieces = initial_symbols(region)
while True:
    candidates = eligible_adjacent_pairs(pieces, ranks)
    if not candidates:
        break
    pair = minimum_rank_pair(candidates, ranks)
    pieces = replace_nonoverlapping_pairs(pieces, pair)
ids = [vocabulary[piece] for piece in pieces]

Suppose bc ranks before ab, both are available, and no larger merge exists. Ranked BPE encodes abc as a|bc. Left-to-right longest matching chooses ab|c. The vocabulary alone therefore does not specify the segmentation decision. [18]

Decoding and text preservation

Tokenizer decoding maps IDs through vocabulary pieces into text. Generation decoding instead selects the next output token; LLM Inference covers that process. Reconstruction cannot recover distinctions that encoding discarded. [21][22]

UTF-8 reconstruction must handle complete byte sequences. The replacement character, U+FFFD, marks decoding errors; it does not preserve their original bytes. Incremental reconstruction retains incomplete trailing bytes until more arrive, rather than treating every token boundary as the end of a string. [5][24]

Text equality and ID equality are separate invariants. In GPT-2's published fixtures, [15, 15] reconstructs 00, whose encoding is [405]. Re-encoding selects the tokenizer's segmentation, not necessarily the sequence originally supplied. [25]

Special tokens and conversation formatting

A special token is a designated vocabulary entry with a structural or control role. Its readable spelling and reserved ID are different representations. A separator marks a boundary; beginning, ending and padding conventions depend on the model. [26]

A chat template serializes ordered messages into model-specific content and boundaries. HTTP request JSON is not necessarily the tokenized text. A generation prompt supplies an assistant-start prefix where required. [29]

Together these contributions produce <s> [INST] Hi [/INST] Hello</s> [INST] Bye [/INST]. This configuration assigns BOS ID 1 and EOS ID 2; the complete conversation IDs are not shown. Its template adds no separate assistant prefix when add_generation_prompt is enabled. That option's effect depends on the template. [30]

Boundary insertion needs one owner. Training with duplicated BOS markers and serving with one changes the representation even when message text matches. Daniel Han's fine-tuning examples illustrate this mismatch. Marker recognition also does not authorize instructions: role formatting cannot by itself separate trusted commands from hostile content; AI Security covers that boundary. [31][32]

Language and formatting effects

Capitalization, leading spaces and surrounding punctuation can change a word's segmentation. Unusual code identifiers can fragment where common names remain compact. These effects follow vocabulary coverage and preprocessing; they do not establish that tokenization alone causes spelling or reasoning failures. [33][34]

Subword fertility is the average number of subwords per tokenized word. Its denominator requires a stated word-segmentation convention. A separate measure records how often words split at all. Neither quantity measures comprehension, and comparisons across writing systems require care about what counts as a word. [36]

Equivalent meanings also need not have equal lengths. A study using 2,000 FLORES-200 sentences translated into 200 languages compared token counts for corresponding translations. Ratios varied across evaluated languages and tokenizers. Topic selection, named entities and translation choices affect those ratios, so representation efficiency must remain distinct from translation quality or general language ability. [37]

Joins, prefixes and offsets

Joining text can create eligible pairs across the join. With the earlier bc-first rules, separately encoding a and bc agrees with encoding abc; separately encoding ab and c does not. Equality must be established for the actual join, not assumed from correct fragment encodings. [18]

An unfinished suffix may change when more text arrives, while pre-tokenization boundaries constrain possible merges. Consequently, tokenized prefixes need not remain prefixes after extension. Truncating IDs, decoding them and re-encoding text can also change segmentation. Explicit structural boundaries and ordinary text joins must be treated according to their configured rules. [28][8]

The same numerical interval selects different text because normalization changes the coordinate system. These internal Rust ranges do not establish every tokenizer API's offset units. Token positions, source-byte offsets and character indexes require explicit conversion rather than direct substitution. [38]

Complete-request token counts

An exact local count describes one fully specified representation. Fix the tokenizer artifacts, preprocessing, special-token policy and template, then count the final IDs. Document-level counts are useful for screening, as demonstrated in Noah Hein's workshop, but do not establish the length of a subsequently assembled conversation. [39][40]

# tokenizer and tools match the checkpoint's template.
ids = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=True,
    add_generation_prompt=True,
)
input_count = len(ids)

Claude's counting endpoint accepts structured messages, system prompts and tool definitions, but documents its result as an estimate that may differ slightly from message creation. A reproducible local count and an eventual provider count therefore make different promises. Fixed character ratios or per-message constants cannot erase that distinction. [42][41]

The context window is a finite token allowance. Known input length, requested maximum output and actual generated length are distinct; their combined constraint and separate limits depend on the model and endpoint. Output caps do not themselves validate input fit. Context Engineering covers selecting and reducing included material when the complete request exceeds its allowance. [43][27]

Tokenizer and model compatibility

A checkpoint is a saved model artifact containing learned state. Its tokenizer must preserve the expected vocabulary mapping and input conventions. A different tokenizer can assign the same integer to a different entry. Matching vocabulary sizes or readable reconstructed text therefore does not establish compatibility. [44][39]

Adding vocabulary entries requires corresponding model dimensions, but resizing alone does not teach useful behavior for new IDs. Tokenizer artifacts, segmentation settings, special-token definitions and the chat template need to move together with the model. [23][45]

# Fixture contract: deterministic, text-preserving settings.
ids = encode(text)
assert decode(ids) == text
assert ids == expected_ids

The assertions establish different properties: preserved text and unchanged representation. After an intentional tokenizer update, changed expectations require review of the responsible configuration. Automatically accepting new fixtures can hide an unintended change. Passing representation checks still does not establish model quality. [23][36]

Open questions

1. 01

   Reducing unequal representation costs across languages remains difficult because word boundaries, translations and corpus composition change the comparison. Progress would require smaller disparities on several parallel and domain-specific corpora without sacrificing text coverage or downstream quality. [37][36]

2. 02

   Choosing vocabulary granularity under constrained resources remains a joint design problem: larger inventories and longer sequences impose different burdens. A useful comparison would hold data and resource allowances fixed while measuring both representation length and task quality, rather than declaring a winner from token count alone. [11][46]

3. 03

   Portable source-span alignment remains difficult when normalization changes byte lengths and APIs expose different coordinate units. Progress would be an explicit cross-runtime contract that reproduces the same original spans for composed, decomposed and compatibility-normalized text. [38]