The Tokenizer

A language model receives integers. The sentence you type has to cross a boundary before any attention layer, embedding lookup, or matrix multiplication can start. The tokenizer does that conversion: it cuts text into vocabulary pieces, then swaps every piece for an ID.

Where those cuts land shapes everything downstream. They set how much text fits in a context window, which strings get compact learned representations, what your input costs, and where spelling, code, numbers, names, and underrepresented languages start getting awkward. Two models can read the same visible sentence as completely different integer sequences, because their tokenizers learned different vocabularies from different corpora.

This drawing follows byte-pair encoding from vocabulary training through deterministic encoding, byte fallback, embedding lookup, systems cost, and the failures that follow from all of it. It closes on the design boundary between BPE, WordPiece, Unigram, bytes, and tokenizer-free models. The lesson underneath is mechanical: a token is a trained allocation decision, trading vocabulary space against sequence length.

Computers store text as bytes. A transformer wants a bounded sequence of indices into a learned table, which is a different thing, and every obvious unit fails somewhere. One token per byte gives you complete coverage and a tiny base vocabulary while making ordinary text very long. One token per Unicode character sits closer to writing, until you remember Unicode contains scripts, combining marks, emoji sequences, and compatibility forms that make "one visible character" a much slipperier idea than it sounds.

Words look efficient right up until the language moves. Inflection splits walk, walked, and walking into three entries. Product names, misspellings, URLs, source identifiers, and terms coined last Tuesday grow the inventory with no upper bound. Languages that don't delimit words with whitespace break the assumption immediately. A fixed word vocabulary has to either emit an unknown token or drag around an impractically large tail of rare entries.

Subword tokenization takes the middle. Frequent strings become single tokens. Rare strings decompose into reusable pieces, characters, or bytes. Tokenization might come apart into token + ization; a name nobody's heard of might become several smaller fragments. Coverage stays finite, and common text still comes out shorter than its character or byte representation.

Vocabulary size is where two costs meet. A larger vocabulary adds rows to the embedding and output matrices. A smaller one produces more positions, which eats context and adds work across the sequence. Tokenizer design is choosing a point on that curve for one particular corpus, model, and serving budget.

Byte-pair encoding started life as compression. Philip Gage's 1994 algorithm repeatedly replaced the most common adjacent byte pair with a byte that didn't appear in the data, storing the substitutions so the stream could be rebuilt. The subword version keeps the repeated-pair idea and treats each selected pair as a new vocabulary symbol instead.

Training starts by writing every corpus word as base symbols and keeping its frequency. Count every adjacent pair, weighted by how often its word shows up. Merge the pair with the largest count everywhere it appears. Add the joined symbol to the vocabulary, recount, repeat, and stop when the vocabulary hits its target size or another rule fires.

Say low, lower, and lowest all occur often. Early counts might pick l + o → lo, then lo + w → low, and later merges get to use low as a single symbol. A different corpus might have gone for e + r first and ended up somewhere else entirely. Ties need a deterministic convention too. What the merge order records is the exact path taken through all those choices.

WordPiece comes from the same subword lineage and doesn't use raw pair frequency as its selection rule. The system Schuster and Nakajima described for Japanese and Korean voice search optimized its wordpiece inventory around language-model likelihood instead. Later WordPiece tokenizers usually encode with greedy longest matching, marking continuation pieces like ##ing.

GPT-2 went with a byte-level BPE variant^[2]. It maps all 256 possible bytes into visible Unicode symbols, then learns merges over that alphabet. The base inventory ends up complete for arbitrary byte strings, and frequent sequences still get to collapse into larger tokens.

The frequency toggle in that bench exposes something easy to miss: duplication changes the tokenizer. Double one word and its internal pairs get more votes in every round. One changed winner early on creates different symbols for every round after it, so two merge tables can diverge substantially even when almost all of the corpus text is identical.

Vocabulary training happens offline, once. After that the tokenizer is frozen: encoding new text doesn't count pairs again or learn anything from your prompt. It applies the stored preprocessing rules and replays the merge priorities it already has.

Start with the base symbols for one region of input. Find the pairs that appear in the merge table. Apply the eligible pair with the earliest rank, update the symbol sequence, and keep going until no ranked pair is left. Real implementations do this far more efficiently than the description suggests, and the rank order still defines the answer.

Order matters because one merge builds the input to another. With ranks e + r → er and then low + er → lower, the second operation can't happen before er exists. Treat the vocabulary as an unordered bag of strings and greedily grab whatever long piece is available, and you'll produce segmentations the trained algorithm never defined.

Determinism covers more than the merges. Unicode normalization, case folding, pattern-based pre-tokenization, whitespace handling, which special tokens are allowed, byte conversion — all of it is tokenizer configuration. Change any one and visually identical text can arrive at the merge stage as different symbols. Version the tokenizer files right next to the model weights, and use the model's own official encoder rather than something that looks equivalent.

Decoding runs it backward: IDs to byte or string pieces, concatenate, convert bytes back to text. An individual byte-level token won't always decode into valid text on its own, since a token boundary can land in the middle of a multi-byte UTF-8 sequence. The full sequence still decodes correctly.

For a bigger side-by-side lab covering character, word, BPE, and WordPiece with prepared examples, IDs, and pricing, use the interactive tokenization demo. This drawing stays on the learned merge circuit.

Unicode assigns numbers — code points — to characters and symbols, and UTF-8 serializes each code point into one to four bytes. ASCII letters take one. Many accented letters take two. Common CJK characters usually take three, and plenty of emoji take four before variation selectors, skin-tone modifiers, or zero-width joiners get involved.

A byte-level tokenizer starts with coverage for all 256 byte values, so any UTF-8 text can fall back to bytes when no larger learned merge matches. At the byte layer there's no such thing as an out-of-vocabulary string. What you pay instead is fragmentation: an uncommon character can burn several tokens if its bytes never got merged together during vocabulary training.

Whitespace is data, not formatting. GPT-style vocabularies are full of entries whose decoded form starts with a space, and SentencePiece uses a visible ▁ marker for a word boundary. That's why token at the start of a string can get a different ID from token after a space. Newlines, tabs, repeated spaces, and indentation each move boundaries around.

Normalization can merge or separate forms that look identical on screen. The character é can be one precomposed code point, or e followed by a combining acute accent. A normalizer might make those the same before tokenization; a tokenizer without it sees two different byte sequences. Anything security-sensitive or code-preserving needs to know precisely which transformations run.

Emoji and CJK fragmentation is a corpus effect sitting on top of UTF-8. A tokenizer trained mostly on English bytes collects lots of votes for English letter sequences and far fewer for anything else. In a multilingual corpus, common CJK byte sequences can still earn full-character or multi-character tokens. Rare emoji sequences tend to split across code-point components or bytes. Byte fallback guarantees you can encode anything; it guarantees nothing about doing it efficiently.

Once segmentation is done, the tokenizer maps each vocabulary piece to an integer ID. Those numbers are arbitrary addresses and nothing more. ID 42 isn't smaller, simpler, or more closely related to ID 43 than it is to ID 40,000. Shuffle the ID assignment and shuffle the model's corresponding rows to match, and the computation comes out identical.

The model's first operation is an embedding lookup. With vocabulary size V and model width d, the learned matrix E has shape V × d, and token ID i selects row E_i. Position information and whatever else the model needs get combined with those row vectors before any transformer block touches the sequence.

That's the bridge into the Transformer's token and embedding path in §02. Worth sitting with: the transformer never sees a token's spelling alongside its ID. Character-level relationships have to be inferred through training, spread across multiple tokens, or supplied by an architecture that takes finer-grained inputs.

The same boundary leads into the embedding space. A transformer embedding row represents one vocabulary token before any context reaches it. A text embedding model later pools contextual token states into a vector for a sentence, a passage, or some other object. Tokenization decides which pieces that encoder gets and how many positions it has to pool.

Special tokens hold reserved IDs. End-of-text, beginning-of-sequence, padding, separators, chat-role markers, tool delimiters — all structure the input without corresponding to ordinary visible text. Their IDs and behavior are model contracts, not conventions. Pass a special token as literal text and it might be rejected, escaped into ordinary pieces, or read as control syntax, depending entirely on the API.

Context limits count tokens, not characters and not words. A 100,000-token window holds wildly different amounts of visible text depending on the language, the code density, the whitespace, and the tokenizer. Instructions, conversation history, retrieved passages, tool schemas, the current query, and the generated output all draw down the same position budget.

Tokens track work too. Processing a prompt means computing representations across every input position: attention in a standard dense layer compares positions against positions, while feed-forward work grows linearly with the position count. Generating autoregressively means another full model pass per token, each one extending the key-value cache. More tokens generally means more latency and more accelerator work, though batching, kernels, caching, and hardware make any universal milliseconds-per-token rule useless.

Most APIs price input and output per token, usually at different rates, and the arithmetic is exactly what it looks like. Cached-input discounts, batch tiers, and model-specific rates complicate the invoice without changing the tokenizer's role in deciding what gets counted.

Fertility measures how many tokens a tokenizer emits per word, or per some other comparable unit of text^[3]. Lower fertility means that language packs more compactly. Comparing across scripts takes care, since whitespace-delimited words aren't universal — parallel translations, characters, bytes, or linguistically segmented units can all give a fairer denominator depending on the study.

A vocabulary trained on uneven data allocates its scarce whole tokens and merges unevenly, which is the polite way of describing it. Languages with fewer or less varied training examples need more pieces to say the same thing. The result is a systems inequity with three components: less text fits, requests cost more, and any useful relationship has to reach across a longer token sequence.

Spelling and letter counting is where the abstraction mismatch surfaces. The model might see straw + berry, one token for the whole word, or a handful of byte-derived fragments. None of those inputs says anywhere that there are three r characters in there. The model can learn spelling patterns statistically, and it can use a tool or take deliberate intermediate steps, but direct access to letter positions simply isn't in the input when a whole fragment arrives as one ID.

Arithmetic hits a related boundary problem. Digits can group in twos, threes, or longer frequent strings, and the grouping shifts depending on a leading space or neighboring punctuation. So the model has to learn arithmetic across several different surface segmentations of the same numeric structure. Tokenization doesn't explain every arithmetic error by itself. Inconsistent digit chunks definitely raise the learning burden.

Code inherits fragmentation from both words and punctuation. A common keyword takes one token while customerInvoiceAccumulator takes many. Indentation and newlines consume positions of their own. Rename an identifier and sequence length and boundaries shift across the whole file without the program behaving any differently. Models trained heavily on code often get code-specific merges, spending vocabulary capacity to shorten frequent operators, syntax, and identifier fragments.

Names live out in the long tail. A frequent public surname might have its own entry, while a rare personal or product name decomposes into fragments that carry associations with unrelated common strings. That makes exact copying, pronunciation, and telling two entities apart harder than it should be. Byte fallback preserves the spelling in principle; the model still has to carry those pieces accurately all the way through generation.

Multilingual inequity is a failure line and a resource allocation choice at the same time. Higher fertility burns through context sooner and makes attention span more positions for the same content, and training supplies fewer repetitions for rare fragments on top of that. Tokenizer coverage, corpus balance, and model capacity all interact, so no token count on its own proves a quality gap. What it does prove is one measurable source of unequal cost and sequence length.

All of these failures respond to the same diagnostic method: inspect the boundaries with the correct tokenizer, compare token counts across representative inputs, and test the exact operation you care about. A model that fumbles strawberry may copy unfamiliar names perfectly well. A model with compact Japanese tokenization may still have no Japanese domain data. Boundaries tell you where the pressure is. They don't settle the causal story.

BPE, WordPiece, and Unigram are all hunting for reusable subwords, and their training and encoding rules diverge. BPE builds symbols through a sequence of pair merges and replays their ranks. WordPiece construction is associated with improving a language-model objective, then usually encodes by greedy longest match. Unigram goes the other direction: start from a large candidate inventory, assign piece probabilities, and prune candidates repeatedly while keeping the likely segmentations.

Taku Kudo's subword regularization work^[4] surfaced another benefit of a probabilistic tokenizer. Training can sample among multiple valid segmentations instead of showing the same fixed boundary every single time, so the model learns that nearby segmentations mean the same text. At inference, a best-scoring segmentation can still be deterministic.

SentencePiece, from Kudo and John Richardson^[5], is a tokenizer toolkit rather than a single segmentation algorithm. It supports BPE and Unigram models and trains straight from raw sentences. Treating whitespace as a normal symbol drops the dependence on an external language-specific word splitter and makes detokenization from the emitted pieces straightforward.

Byte-level models push the boundary further down. ByT5 processes UTF-8 bytes directly^[6] and reported competitive results across a range of multilingual tasks, holding up especially well on noisy input. The bill comes due in sequence length. Dropping a large subword vocabulary saves parameters in the embedding and output layers, and now transformer computation stretches across many more positions.

Token-free and character-level research keeps going. CANINE works on character sequences with downsampling to keep the sequence length manageable^[7]. Other designs learn segmentation inside the network, or compress byte and character patches dynamically. All of them reduce dependence on a frozen language-specific vocabulary, and all of them relocate the complexity into longer inputs, downsampling, local encoders, or learned patching.

There's no context-free winner here. A larger subword vocabulary shortens well-represented text and inflates the token-dependent matrices. A smaller one improves base coverage and parameter efficiency while stretching sequences out. Multilingual breadth competes for entries against domain-specific compactness. Code, DNA, speech units, and ordinary prose all have different frequency structures, and one vocabulary has to pick.

Changing a vocabulary also breaks compatibility, which surprises people. Adding one attractive token after pretraining gives it no useful embedding row or output weight, and shifting existing IDs points the model at the wrong learned vectors entirely. Extending a tokenizer means coordinated model training, initialization, evaluation, and usually changes to serving artifacts too. You can sometimes reserve unused IDs in advance or fine-tune newly added rows — either way it's a model migration, not a harmless text-processing update.

Match the training corpus to the traffic mix the system is supposed to serve. Raw frequency on its own will happily let duplicated boilerplate consume merges that would have been worth far more elsewhere. Corpus sampling, per-language weighting, normalization, and vocabulary budget are all expressing product priorities whether or not anyone says so. Evaluate that allocation before expensive model training starts, because fertility statistics and boundary inspection surface the obvious gaps while tokenizer experiments are still cheap.

The decision belongs next to the architecture and data plan:

• Measure fertility on each target language and domain.
• Include normalization, whitespace, and special-token behavior in the versioned contract.
• Price embedding/output matrix parameters against sequence compute and cache memory.
• Test character-sensitive, numeric, code, name, and noisy-text cases directly.
• Keep the official tokenizer artifact inseparable from model deployment.

The tokenizer is the first learned interface in the whole circuit. It decides which strings earn a single address, which have to be assembled out of smaller units, and how much sequence is left for the rest of the model. Every later layer works inside that allocation.