The Tokenizer

A language model receives integers. The sentence you type has to cross a boundary before any attention layer, embedding lookup, or matrix multiplication can begin. A tokenizer performs that conversion: it divides text into vocabulary pieces, then replaces every piece with an ID.

Those boundaries shape the whole system. They determine how much text fits in a context window, which strings receive compact learned representations, how much input costs, and where spelling, code, numbers, names, and underrepresented languages become awkward. Two models can read the same visible sentence as different integer sequences because their tokenizers learned different vocabularies from different corpora.

This drawing follows byte-pair encoding from vocabulary training through deterministic encoding, byte fallback, embedding lookup, systems cost, and characteristic failures. It closes with the design boundary between BPE, WordPiece, Unigram, bytes, and tokenizer-free models. The central lesson is mechanical: a token is a trained allocation decision between vocabulary space and sequence length.

Computers can store text as bytes, but a transformer expects a bounded sequence of indices into a learned table. The obvious units all fail in different ways. One token per byte gives complete coverage and a tiny base vocabulary, while making ordinary text long. One token per Unicode character is closer to writing, but Unicode contains scripts, combining marks, emoji sequences, and compatibility forms that complicate the idea of one visible character.

Words look efficient until language starts moving. Inflection turns walk, walked, and walking into separate entries. Product names, misspellings, URLs, source identifiers, and newly coined terms grow the inventory without limit. Languages without whitespace-delimited words expose the assumption immediately. A fixed word vocabulary must either emit an unknown token or carry an impractically large tail of rare entries.

Subword tokenization chooses a middle point. Frequent strings can become single tokens. Rare strings decompose into reusable pieces, characters, or bytes. Tokenization might become token + ization; an unfamiliar name might become several smaller fragments. Coverage stays finite while common text remains shorter than its character or byte representation.

Vocabulary size therefore connects two costs. A larger vocabulary adds rows to the embedding and output matrices. A smaller vocabulary produces more positions, increasing context use and the work performed over the sequence. Tokenizer design picks a point on that curve for a particular corpus, model, and serving budget.

Byte-pair encoding began as compression. Philip Gage's 1994 algorithm repeatedly replaced the most common adjacent byte pair with a byte that did not occur in the data, storing substitutions so the stream could be reconstructed. The subword version keeps the repeated-pair idea but uses each selected pair as a new vocabulary symbol.

Training starts by expressing every corpus word as base symbols and retaining its frequency. Count every adjacent pair, weighted by how often its word occurs. Merge the pair with the largest count everywhere it appears. Add the joined symbol to the vocabulary, recount, and repeat until the vocabulary reaches its target size or another stopping rule fires.

Suppose low, lower, and lowest occur often. Early counts may select l + o → lo, then lo + w → low. Later merges can use low as one symbol. A different corpus might favor e + r first. Frequency ties also need a deterministic convention. The resulting merge order records the path taken through these choices.

WordPiece belongs to the same subword lineage but does not use raw pair frequency as its defining selection rule. The system described by Schuster and Nakajima for Japanese and Korean voice search optimized a wordpiece inventory around language-model likelihood. Later WordPiece tokenizers commonly encode with greedy longest matching, marking continuation pieces such as ##ing.

GPT-2 used a byte-level BPE variant^[2]. It maps the 256 possible bytes into visible Unicode symbols, then learns merges over that alphabet. This makes the base inventory complete for arbitrary byte strings while still allowing frequent sequences to collapse into larger tokens.

The frequency toggle exposes a training fact that is easy to miss: duplication changes the tokenizer. Double one word and its internal pairs receive more votes at every round. An early changed winner creates different symbols for later rounds, so the merge tables can diverge even when nearly all corpus text stays the same.

Vocabulary training is an offline operation. Once the tokenizer is fixed, encoding new text does not count pairs again or learn from the prompt. It applies the stored preprocessing rules and replays the learned merge priorities.

Start with the base symbols for one input region. Find pairs that appear in the merge table. Apply the eligible pair with the earliest rank, update the symbol sequence, and continue until no ranked pair remains. Equivalent implementations can process the same operation more efficiently, but the rank order still defines the result.

Order matters because one merge creates the input to another. Given ranks e + r → er, then low + er → lower, the second operation cannot exist before er has been formed. Treating the vocabulary as an unordered bag of strings and greedily taking any available long piece can produce a segmentation that the trained algorithm never defined.

Determinism is broader than merges. Unicode normalization, case folding, pattern-based pre-tokenization, whitespace handling, allowed special tokens, and byte conversion all belong to the tokenizer configuration. Change one and identical visible text may reach the merge stage as different symbols. Production systems should version the tokenizer files beside the model weights and use the model's official encoder.

Decoding reverses IDs to their byte or string pieces, concatenates them, and converts bytes back into text. Individual byte-level tokens do not always decode into valid text by themselves because a token boundary can split a multi-byte UTF-8 sequence. The complete sequence can still decode correctly.

For a larger side-by-side lab covering character, word, BPE, WordPiece, prepared examples, IDs, and pricing, use the interactive tokenization demo. The drawing here stays focused on the learned merge circuit.

Unicode assigns numbers called code points to characters and symbols. UTF-8 serializes each code point into one to four bytes. ASCII letters use one byte. Many accented letters use two. Common CJK characters usually use three, and many emoji use four before variation selectors, skin-tone modifiers, or zero-width joiners enter the sequence.

A byte-level tokenizer begins with coverage for all 256 byte values. Any UTF-8 text can therefore fall back to bytes when no larger learned merge matches. There is no out-of-vocabulary string at the byte layer. The tradeoff appears as fragmentation: an uncommon character may consume several tokens if its bytes were never merged together during vocabulary training.

Whitespace is data. GPT-style vocabularies often contain entries whose decoded form begins with a space; SentencePiece uses a visible ▁ marker for a word boundary. That is why token at the beginning of a string can receive a different ID from token after a space. Newlines, tabs, repeated spaces, and indentation can each alter boundaries.

Normalization can combine or separate visually equivalent forms. The character é can be one precomposed code point or e followed by a combining acute accent. A normalizer may make them identical before tokenization; a tokenizer without that normalization sees different byte sequences. Security-sensitive and code-preserving applications need to know exactly which transformations occur.

Emoji and CJK fragmentation is a corpus effect layered over UTF-8. A tokenizer trained mostly on English bytes sees many votes for English letter sequences and fewer for other scripts. Common CJK byte sequences may still become full-character or multi-character tokens in a multilingual corpus. Rare emoji sequences can split across code-point components or bytes. Byte fallback prevents failure to encode, but it does not promise equal efficiency.

After segmentation, the tokenizer maps every vocabulary piece to an integer ID. The numbers are arbitrary addresses. ID 42 is not smaller, simpler, or more related to ID 43 than to ID 40,000. Rearranging the ID assignment while rearranging the model's corresponding rows would leave the computation unchanged.

The first model operation is an embedding lookup. If the vocabulary has size V and the model width is d, the learned matrix E has shape V × d. Token ID i selects row E_i. Position information and other model-specific inputs are then combined with those row vectors before transformer blocks operate on the sequence.

This is the bridge into the Transformer’s token and embedding path in §02. The transformer never receives the spelling of a token beside its ID. Character relationships must be inferred through training, represented across multiple tokens, or supplied by architectures with finer-grained inputs.

The same boundary leads into the embedding space. A transformer embedding row represents one vocabulary token before context. A text embedding model later combines contextual token states into a vector for a sentence, passage, or other object. Tokenization controls the pieces available to that encoder and the number of positions it must pool.

Special tokens occupy reserved IDs. End-of-text, beginning-of-sequence, padding, separators, chat-role markers, and tool delimiters can structure the input without corresponding to ordinary visible text. Their exact IDs and behavior are model contracts. Passing a special token as literal text may be rejected, escaped into ordinary pieces, or interpreted as control syntax depending on the API.

Context limits count tokens, not characters or words. A 100,000-token window can hold very different amounts of visible text depending on language, code density, whitespace, and the tokenizer. Instructions, conversation history, retrieved passages, tool schemas, the current query, and generated output all draw from that same position budget.

Tokens also track work. During prompt processing, the model computes representations across all input positions. Attention in a standard dense layer compares positions with positions, while feed-forward work grows linearly with position count. During autoregressive generation, each new token requires another model pass and extends the key-value cache. More tokens usually mean more latency and accelerator work, though batching, kernels, caching, and hardware prevent a universal milliseconds-per-token rule.

Many APIs price input and output per token, often at different rates. The arithmetic is direct: token count multiplied by the applicable rate. Cached-input discounts, batch tiers, and model-specific rates complicate invoices, but they do not change the tokenizer's role in determining the counted units.

Fertility measures how many tokens a tokenizer produces per word or another comparable text unit^[3]. Lower fertility packs that language more compactly. Comparing across scripts requires care because whitespace-delimited words are not universal; parallel translations, characters, bytes, or linguistically segmented units can provide fairer denominators for a specific study.

A vocabulary trained on uneven data tends to allocate its scarce whole tokens and merges unevenly. Languages with fewer or more diverse training examples can require more pieces for comparable content. The result is a systems inequity: less text fits, requests cost more, and useful relationships must cross longer token sequences.

Spelling and letter counting expose the abstraction mismatch. A model may see straw + berry, one token for the whole word, or several byte-derived fragments. None of those inputs explicitly says there are three r characters. The model can learn spelling patterns statistically and can use tools or deliberate intermediate steps, but direct access to letter positions is absent when a whole fragment arrives as one ID.

Arithmetic has a related boundary problem. Digits may group in twos, threes, or longer frequent strings, and the grouping can change with leading spaces or neighboring punctuation. The model must learn arithmetic across several surface segmentations of the same numeric structure. Tokenization does not fully explain arithmetic errors, but inconsistent digit chunks raise the learning burden.

Code inherits both word and punctuation fragmentation. A common keyword may occupy one token while customerInvoiceAccumulator becomes many pieces. Indentation and newlines consume positions. An identifier rename can alter sequence length and boundaries throughout a file without changing program behavior. Models trained heavily on code can receive code-specific merges, spending vocabulary capacity to shorten frequent operators, syntax, and identifier fragments.

Names live in the long tail. A frequent public surname may have one entry; a rare personal or product name may decompose into fragments associated with unrelated common strings. That can make exact copying, pronunciation, and entity distinction harder. Byte fallback preserves the spelling in principle, while the model still has to carry the pieces accurately through generation.

Multilingual inequity is both a failure line and a resource allocation choice. Higher fertility consumes context sooner and makes attention span more positions for comparable content. Training also supplies fewer repetitions for rare fragments. Tokenizer coverage, corpus balance, and model capacity interact, so no single token count proves a quality gap; it does identify one measurable source of unequal cost and sequence length.

These failures share a diagnostic method: inspect boundaries with the correct tokenizer, compare token counts across representative inputs, and test the exact operation. A model that fails strawberry may still copy unfamiliar names well; a model with compact Japanese tokenization may still lack Japanese domain data. Boundaries locate pressure. They do not settle the whole causal story.

BPE, WordPiece, and Unigram all seek reusable subwords, but their training and encoding rules differ. BPE builds symbols through a sequence of pair merges and replays their ranks. WordPiece vocabulary construction is associated with improving a language-model objective, then commonly uses greedy longest-match encoding. Unigram starts from a large candidate inventory, assigns piece probabilities, and repeatedly prunes candidates while preserving likely segmentations.

Taku Kudo's subword regularization work^[4] showed another benefit of a probabilistic tokenizer: training can sample multiple valid segmentations instead of presenting one fixed boundary every time. The model learns that nearby segmentations express the same text. At inference, a best-scoring segmentation can remain deterministic.

SentencePiece, developed by Kudo and John Richardson^[5], is a tokenizer toolkit rather than one segmentation algorithm. It supports BPE and Unigram models and trains directly from raw sentences. Treating whitespace as a normal symbol removes dependence on an external language-specific word splitter and supports detokenization from the emitted pieces.

Byte-level models move the boundary down. ByT5 processes UTF-8 bytes directly^[6] and reported competitive results on a range of multilingual tasks, with particular strength under noisy inputs. The cost is longer sequences. Removing a large subword vocabulary saves parameters in the embedding and output layers, but transformer computation now spans many more positions.

Token-free and character-level research pushes further. CANINE operates on character sequences with downsampling to control sequence length^[7]. Other designs learn segmentation inside the network or compress byte/character patches dynamically. These approaches can reduce dependence on a frozen language-specific vocabulary, while moving complexity into longer inputs, downsampling, local encoders, or learned patching.

There is no context-free winner. A larger subword vocabulary shortens well-represented text and enlarges token-dependent matrices. A smaller vocabulary improves base coverage and parameter efficiency while increasing sequence length. Multilingual breadth competes for entries with domain-specific compactness. Code, DNA, speech units, and ordinary prose have different frequency structures.

Vocabulary changes also break compatibility. Adding one attractive token after pretraining does not give it a useful embedding row or output weight, and shifting existing IDs points the model at the wrong learned vectors. Extending a tokenizer requires coordinated model training, initialization, evaluation, and often changes to serving artifacts. Teams can sometimes reserve unused IDs or fine-tune newly added rows, but the result is a model migration rather than a harmless text-processing update.

The training corpus should match the traffic mix the system is expected to serve. Raw frequency alone can let duplicated boilerplate consume merges that would be more valuable elsewhere. Corpus sampling, per-language weighting, normalization, and vocabulary budget jointly express product priorities. Evaluate the resulting allocation before expensive model training, since fertility statistics and boundary inspection can reveal obvious gaps while tokenizer experiments are still cheap.

The decision belongs beside the architecture and data plan:

• Measure fertility on each target language and domain.
• Include normalization, whitespace, and special-token behavior in the versioned contract.
• Price embedding/output matrix parameters against sequence compute and cache memory.
• Test character-sensitive, numeric, code, name, and noisy-text cases directly.
• Keep the official tokenizer artifact inseparable from model deployment.

The tokenizer is the first learned interface in the language-model circuit. It decides which strings earn one address, which must be assembled from smaller units, and how much sequence the rest of the model receives. Every later layer works within that allocation.