The Transformer

In 2017, eight researchers at Google published a paper whose central claim sounded like a provocation: you can throw away recurrence entirely — the thing every serious sequence model had been built on for two decades — and replace it with nothing but attention. The paper was called Attention Is All You Need^[1], and the title turned out to be an understatement. Nine years later, the architecture it introduced is effectively the only one that matters. Language, code, images, audio, protein folding — all of it now runs through the same machine.

What makes that strange is how small the machine is. Strip away the training infrastructure and the trillion tokens of data, and the transformer is about a page of math. Three learned projections, a dot product, a softmax, a two-layer neural network, and a normalization trick — stacked in a loop. That's the entire circuit inside GPT-5, Claude, and Gemini.

This drawing takes the machine apart piece by piece. By the bottom of the page you will know what a token actually is, why attention is just three matrix multiplications, what the 96 layers of a large model are doing all day, and what the single knob called temperature actually trades away. No hand-waving. Every part gets a schematic.

Before 2017, sequence models were recurrent. An RNN reads a sentence the way you read a ticker tape: one token per clock tick, compressing everything seen so far into a single fixed-size hidden state. That design has two fatal flaws. First, memory: by token 500, whatever happened at token 3 has been squeezed through 497 lossy updates. Second, and worse for hardware: it's sequential. You cannot compute step 500 before step 499. GPUs, whose entire value is doing thousands of things at once, sit mostly idle.

The transformer arrived through machine translation, where both flaws were easy to see. Sutskever, Vinyals, and Le's 2014 sequence-to-sequence system^[2] used one recurrent network to compress a source sentence and another to unfold a translation. Reversing the source sentence helped shorten some dependency paths, but the encoder still had to squeeze the whole source through its final state. Bahdanau, Cho, and Bengio then let the decoder consult all encoder states through a learned alignment mechanism^[3]. Luong, Pham, and Manning refined that family of attention functions in 2015^[4]. Attention began as a relief valve attached to recurrence.

Attention Is All You Need made a subtraction claim. The useful alignment mechanism could carry the whole architecture; recurrent and convolutional sequence processing could leave. The published transformer still had an encoder for reading the source and a decoder for producing the target. Its novelty was the route between and within those stacks: every position could exchange information through attention without waiting for a recurrent clock.

The transformer's move is to delete the ticker tape. Every token gets to look directly at every other token, in parallel, in a single matrix multiplication.

The cost is quadratic — n tokens means n² pairwise interactions — but quadratic work that a GPU can do all at once beats linear work it has to do one step at a time. That single trade, more math for less waiting, is why transformers won.

That comparison needs a hardware footnote. Attention performs O(n²) pairwise score work and stores an n-by-n pattern during ordinary training, so very long sequences become expensive fast. Recurrence performs only O(n) state updates, yet those updates form a dependency chain. For the sentence lengths common in early translation work, dense matrix multiplication kept the GPU busier and shortened the path between any two positions from as many as n steps to one attention hop.

Training and generation expose different sides of this design. During next-token training, the entire known sequence is available. A triangular causal mask hides future positions, so the model can compute a prediction at every position in parallel without seeing the answer to its right. During generation, the next token genuinely does not exist yet. The model must append one token, then repeat, which restores a serial outer loop even though each pass uses parallel matrix operations inside.

A transformer never sees words. It sees tokens — chunks from a fixed vocabulary of 50,000–200,000 pieces, learned by greedily merging the most frequent character pairs in the training data (byte-pair encoding). Common words survive as single tokens; rare ones get shattered. the is one token. transformer might be transform + er. This is why models are weirdly bad at counting letters: they can't see them.

Each token ID then indexes into an embedding table — a giant learned matrix with one row per vocabulary entry. Out comes a vector of maybe 512 to 12,288 numbers (the model's width, d_model). Nothing about these numbers is hand-designed; training nudges them until geometry encodes meaning. Directions in this space become semantic: famously, king − man + woman lands near queen. From here on, the entire model is just operations on lists of these vectors.

Tokenization fixes the model's vocabulary before training begins. A frequent fragment earns its own row in the table; an uncommon name may consume several positions and several rounds of computation. Two strings that look almost identical to a person can split differently because capitalization, whitespace, or a leading byte changes the merge path. Token count therefore affects latency, context use, and price as well as spelling behavior.

Here is the part worth slowing down for. Attention answers one question for every token: which other tokens should influence my meaning, and by how much? The mechanism is a soft dictionary lookup. Each token emits three vectors, produced by three learned projection matrices: a query (what am I looking for?), a key (what do I contain?), and a value (what do I contribute if selected?).

To update token i, take its query and dot it against every token's key. High dot product means "relevant." Divide by √d_k so the numbers don't blow up as vectors get wide, softmax the scores into weights that sum to 1, and use those weights to blend everyone's value vectors into a new representation of token i. That's it. That is the mechanism the trillion-dollar industry runs on:

Queries and keys decide routing; values carry payload. Imagine looking up a library book. Your request is the query, catalog entries are keys, and the pages returned are values. A close catalog match should route the pages to you, yet you do not want the catalog text itself as the answer. Learned projections let the model separate those roles even though all three start from the same residual-stream vector.

Abstract until you watch it work. Below is the canonical sentence from the Transformer's release. The pronoun "it" is ambiguous — it could be the animal or the street. Hover any token to see where its attention goes. Hover it: trained models learn, from nothing but text statistics, that tired things are animals, not streets. The weights below are hand-drawn to match the published visualizations.

One attention pattern per layer is too few. "Which noun does this pronoun bind to?" and "what verb governs me?" are different questions with different answers. So the transformer runs attention several times in parallel — 8 to 128 heads — each with its own small W_Q, W_K, W_V, each free to learn its own notion of relevance. Their outputs are concatenated and mixed back down with one more matrix.

This is nearly free: each head works in d_model/h dimensions, so the total compute matches single-head attention. When researchers dissect trained models they find heads with legible jobs — previous-token heads, syntax heads, induction heads that scan for "the last time this pattern appeared, what came next?" Induction heads are the leading candidate explanation for in-context learning.

The job labels are descriptions of observed behavior, not slots assigned by an engineer. Training can distribute one behavior across several heads, reuse a head for different patterns, or leave a head with no clean human story. A head also works inside a layer: an early head may copy local punctuation while a later head routes a representation already enriched by dozens of earlier operations. Reading one attention map never reveals the whole computation.

There's a bug in everything above. Attention is a sum over all positions — it treats the input as a bag of tokens. Reorder "dog bites man" into "man bites dog" and attention produces the same output vectors, just reordered: the mechanism cannot tell which order the words came in. The fix: stamp each token with a vector encoding its position before the first layer. The original paper used interleaved sine and cosine waves at geometrically spaced frequencies — fast oscillations in the early dimensions, glacial ones in the late dimensions, giving every position a unique barcode:

Why sinusoids instead of the raw integer position? Because for any fixed offset k, PE(pos+k) is a linear function of PE(pos) — the same trigonometric identity that makes clock arithmetic work — so "three tokens back" becomes something a learned matrix can express. Modern models mostly use a descendant called RoPE, which rotates the query and key vectors by position instead of adding to them, but the idea is identical: order enters the model as geometry.

Position methods also define how a model behaves near and beyond its trained context length. Learned absolute position tables have a fixed set of rows. Sinusoids can be evaluated at new positions, although availability does not guarantee useful extrapolation. RoPE makes attention scores depend naturally on relative displacement, but extending its range still requires training choices or frequency adjustments. A large advertised context window is an empirical capability, not a consequence of having enough position numbers.

Attention alone can't compute much — it only mixes information between positions. So each block pairs it with a feed-forward network: two plain linear layers with a nonlinearity between them, applied to every token independently, usually 4× wider than the model itself. If attention is the routing layer, the FFN is the processing plant — it holds roughly two-thirds of the parameters, and interpretability work suggests much of the model's factual knowledge lives here, as key-value memories.

Two pieces of plumbing hold the tower together. Residual connections: every sub-layer's output is added to its input rather than replacing it, so the block computes a correction, not a rewrite — and gradients get a clean highway from layer 96 back to layer 1. Layer normalization re-centers each vector before every sub-layer so activations can't drift off to infinity across dozens of layers. Neither is glamorous; without both, deep transformers simply don't train.

The residual stream explains how specialized operations cooperate. Each attention or feed-forward sublayer reads the current vector and writes an update into the same shared width. Information can persist when a sublayer writes almost nothing, and several sublayers can accumulate compatible features without handing a bespoke data structure from one to the next. Interpretability researchers can often trace a behavior as a sequence of reads and writes along this stream.

That's the entire architecture. A GPT-class model is: embedding table → N copies of this block → one final projection back to vocabulary size. Each layer reads the residual stream, adds its two cents, and passes it up. By the top, "it" doesn't mean a pronoun anymore — it means the tired animal, subject of the next clause.

The original translation transformer used both encoder and decoder stacks. GPT took the decoder side, kept causal self-attention, and trained it on one plain objective: predict the next token across large unlabeled text. GPT-1 showed in 2018 that generative pretraining could be adapted to varied language tasks^[5]. GPT-2 enlarged the model and dataset in 2019^[6]; GPT-3 enlarged them again in 2020 and demonstrated that examples written directly in the prompt could steer behavior without a gradient update^[7].

The final layer outputs one raw score (logit) per vocabulary entry. A softmax turns those into a probability distribution, and generation is just sampling from it, appending the winner, and running the whole stack again — one forward pass per token. The knob everyone actually touches is temperature T, which divides the logits before the softmax. Low T sharpens the distribution toward the single best guess; high T flattens it toward chaos. Drag it:

This is also where the model's honest epistemology lives: it never outputs a word, only a distribution over words. Sampling is the function — and a fluent draw from that distribution can land on unsupported content under any decoding policy, greedy included. That failure is what gets called hallucination. Every trick you've heard of (top-k, top-p, beam search) is just a different policy for picking from this same bar chart.

Greedy decoding always selects the highest-probability token. Top-k first discards everything outside the k highest scores; top-p keeps the smallest set whose cumulative probability crosses p. These policies control the candidate set and its randomness. They do not check a database, verify a citation, or turn probability into truth. Reliability needs evidence, tools, constrained output, or checking elsewhere in the system.

The KV cache avoids repeating most work inside that serial loop. Keys and values for previous positions stay fixed at a given layer, so generation stores them and computes only the new position's projections. The new query still compares with all cached keys, which means per-token attention work and cache reads grow with context length. For a batch of long conversations, cache memory can limit serving capacity before parameter memory does.

Sampling settings also interact with the prompt. A low temperature can make repeated runs stable while preserving a confidently wrong continuation. A high temperature can expose several plausible phrasings without adding knowledge. If an application needs one exact enum or JSON shape, constrain the output grammar. If it needs a current fact, supply a current source. Temperature solves neither contract.

Here is the uncomfortable secret of the last nine years: almost nothing in this drawing has changed since 2017. GPT-4 is the same circuit as the original transformer, with bigger matrices, more layers, and vastly more data. Parameters grow roughly as 12·N·d², and the field discovered that loss falls as a smooth, boringly reliable power law in compute. Capability was purchased, order of magnitude by order of magnitude.

Kaplan and collaborators quantified that thread in 2020^[8]: over the regimes they measured, language-model loss followed power-law relationships with model size, dataset size, and compute. The curves gave teams a planning instrument. Run smaller experiments, fit the trend, and estimate what a much larger training run might buy before spending the full budget.

Hoffmann and collaborators revisited the allocation in 2022 with Chinchilla^[9]. Their result was operational: many large models had too many parameters for the number of training tokens they received. Under a fixed compute budget, a smaller model trained on substantially more data could win. Scale therefore has at least three coupled dials — parameters, training tokens, and compute — and the best allocation depends on the budget and target.

The refinements since — RoPE, RMSNorm, SwiGLU, grouped-query attention, mixture-of-experts — are real engineering, but they are pencil revisions in the margins of this sheet, not new sheets. If you understand the drawing above, you understand the machine that is currently rewriting the economics of knowledge work. It fits on one page. It always did.