The Inference Engine

Behind a language model endpoint sits a resource allocator, and most of what people find surprising about inference latency traces back to it. Every request that gets accepted wants a share of four things: the model weights, scratch space for the arithmetic of a single pass, a KV cache — the running notes the model keeps on every token so far, so it never has to recompute them — and repeated access to the chip and its memory bandwidth. The inference engine decides when each request gets in, where its state lives, which other requests it shares an iteration with, and whether a brand-new prompt is allowed to delay tokens already streaming to somebody else.

This drawing follows one request through, then turns on contention. Serving splits into two phases with completely different shapes. Prefill is the model reading your prompt: it can work on every prompt position at once, so raw compute is the limit. Decode is the model writing its answer one token per pass, each pass re-reading the weights, so memory bandwidth is the limit. Those shapes differ enough that a schedule which raises total tokens per second can simultaneously make the first word slower, or widen the gaps between later ones. Memory splits the same way: weights are mostly fixed, activations spike during a pass, and KV state grows with every live token.

The citable object here is a worked memory-and-time ledger. Its 7B teaching case uses explicit binary units and keeps every category on its own line. The scheduler panel runs fixed synthetic arrivals with the assumptions printed next to them — it demonstrates an algorithm, it doesn't benchmark hardware. Product shootouts and vendor benchmark tables stay off the permanent sheets, because model revision, traffic shape, kernels, hardware, and latency targets each change the answer.

You can describe an offline model call as tensor operations and be done. A service has to admit requests, queue them, cancel them, stream them, meter them, isolate them, and recover from them. It takes text or token IDs, applies the chat template and tokenizer, checks the length, picks a replica, waits for capacity, allocates KV pages, runs prefill, samples the first token, and repeats decode until something tells it to stop. Only then can it hand the request's cache blocks back.

State lives at each of those boundaries. The router knows the tenant and the deadline. The scheduler knows arrivals, token budgets, and which sequences are resident. The model runner owns the device buffers and the kernels. The sampler owns the decoding parameters and the random state. The streamer owns backpressure and cancellation. A request can die before a single matrix multiplication happens, finish while its batch peers keep going, or lose its client entirely while device work sits queued on its behalf.

The service clock starts well before the GPU clock. Authentication, request parsing, tokenizer work, replica selection, and queue delay can add up to more than the model time on a short prompt. Streaming adds yet another clock at the network boundary, where a token can be computed and still unavailable to the caller because serialization or a slow connection is backed up. So take timestamps at acceptance, at scheduler admission, at prefill start and end, at each token-ready event, at each token-write event, and at release. Those points separate engine delay from transport delay without pretending the user can't feel both.

All of which makes serving its own discipline, separate from model architecture. The Transformer drawing's KV-cache section covers why earlier keys and values can be reused at all. The inference engine takes that reuse and turns it into an allocation and scheduling problem spanning many unrelated sequences. The site's interactive demos index has adjacent tokenization and retrieval experiments; the scheduler below stays on this page so its assumptions sit next to the ledger they belong to.

Prefill eats the whole prompt. Inside each layer, many prompt positions get projected and processed at once. Attention still respects the causal mask, but what reaches the device is large matrix operations with a lot of arithmetic per launch. Prompt length is what drives time-to-first-token, because decode can't emit anything until the prompt's final position has made it through the model.

Decode picks up where prefill stopped. Each active sequence contributes one current token to the iteration, reads back its previous KV entries, produces logits, samples a token, and appends one K vector and one V vector at every layer. The serial dependency is absolute — token t + 1 needs token t and there's no way around it. Batching adds more sequences to the iteration. No scheduler anywhere can parallelize the future positions of a single autoregressive sequence.

That difference is what predicts interference. Admit one long prefill whole and it can hold the runner long enough to widen the decode gaps for every sequence already streaming. Chunked prefill cuts the prompt into bounded pieces of token work so decode iterations can slip in between the chunks. SARATHI^[1] formalized that piggybacking; DistServe^[2] went further and treated prefill and decode as separable phases you can provision against different latency objectives.

Token throughput needs two counters, because input and output tokens are shaped differently. A prefill-heavy service can post an impressive total-token rate while streaming very few tokens to anyone. A decode-heavy service can spend most of its life rereading weights and growing KV for tiny one-token batches. Report input tokens/s and output tokens/s separately, with the request lengths beside them. End-to-end latency scales with requested output too — roughly TTFT plus the sum of the later ITLs — so comparing requests with different completion lengths without bucketing them is mixing workload into what looks like system behavior.

Weight memory is mostly fixed once a replica has loaded. Call P the parameter count and b_w the stored bytes per parameter. Quantized formats bring along scales, zero points, packing, and sometimes a higher-precision subset, so that metadata belongs on the same line instead of vanishing into a slogan about bit width.

Activation and runtime memory moves around with kernels, batch token count, graph captures, temporary outputs, communication buffers, and allocator policy. Measure it at the batch shape you actually intend to run and carry it as M_runtime,peak. Training-style "activations per token" estimates tend to mislead here, because inference kernels fuse operations and reuse storage in ways the training math doesn't anticipate.

KV state scales directly, and the arithmetic is unforgiving. With L layers, H_kv KV heads per layer, head dimension d_h elements per head, S sequence tokens per request, and b_kv bytes per cached element, every single token stores one key and one value at every layer.

The worked case assumes 7,000,000,000 parameters in BF16, no added weight metadata, 32 layers, 8 KV heads, head dimension 128, BF16 KV, 4,096 tokens per resident request, and 16 resident requests. Weights consume 14,000,000,000 bytes = 13.04 GiB. KV consumes 131,072 bytes/token = 128 KiB/token, then 536,870,912 bytes/request = 0.50 GiB/request, then 8.00 GiB for 16 requests. Add a separately budgeted 3.00 GiB runtime peak and a 4.00 GiB safety reserve on a 40 GiB device. The committed-plus-reserved total is 28.04 GiB.

Sequence length in that multiplication means each request's current prompt plus what it has generated so far — not the configured maximum, unless memory really is reserved out to that maximum. For a mixed batch, swap N × S for the sum of live sequence lengths, Σ_r S_r. Paging rounds each request up to a block boundary, so physical allocation works out to Σ_r ceil(S_r / B) × B tokens for block size B. Prefix sharing can push physical occupancy below the naïve sum when several requests point at the same immutable blocks, and copy-on-write hands ownership back apart the moment a branch diverges.

The safety line is deliberate capacity, not unidentified waste. It absorbs measurement error, allocator behavior, and transient shapes you didn't plan for. Pick it from observed peaks and from how much failure you can tolerate, then write it into the workbook by name. When a runtime reserves memory without committing it, record allocator-reserved and tensor-allocated bytes as two numbers. Device telemetry on its own can't tell you which bytes are reclaimable or which request owns them.

Grouped-query attention drops H_kv below the query-head count, which is exactly why the cache formula asks for KV heads rather than total attention heads. The GQA paper^[3] covers converting multi-head checkpoints down to fewer KV heads. Read the ledger inputs off the model configuration, not off the size in its name.

A naïve allocator hands each request one contiguous region sized to its maximum sequence. When the output finishes early, the unused tail is stranded. Allocate and free enough differently sized regions and you end up with plenty of total free memory and no single hole big enough for the next request. Copying a cache to grow it adds memory traffic at exactly the moment the engine is busiest.

PagedAttention^[4] borrows an operating-system idea for KV storage. A request's logical cache gets divided into fixed-size token blocks, and its block table maps logical block numbers onto non-contiguous physical ones. The next token either fills the current block or triggers another allocation; when the request completes, its blocks go back to a shared pool. Only the last block carries ordinary rounding waste. The vLLM paper also works through copy-on-write sharing for common prompt state and for parallel sampling.

Block size is a genuine trade rather than a tuning knob with a right answer. Smaller blocks cut last-block waste and make small growth increments cheaper, at the price of more block-table entries and more allocation operations. Larger blocks keep the metadata simple and can suit the kernels better, while amplifying rounding loss across a population of short sequences. Prefix caching layers a second policy question on top: reusable blocks only save prefill work if the hit rate and the reuse lifetime repay the bytes and the lookup.

Work an example. Three requests with live lengths of 17, 33, and 65 tokens, under 16-token blocks. Logical occupancy is 115 tokens. Physical occupancy is 2 + 3 + 5 = 10 blocks, or 160 token slots, which leaves 45 slots idle across their last blocks. Drop to four-token blocks and physical occupancy falls to 124 slots while block-table entries climb from 10 to 31. That arithmetic covers rounding and nothing else — kernel layout, allocator metadata, and shared prefixes each add their own term.

Paging improves allocation efficiency. It doesn't reduce the logical KV bytes the formula demands. When pressure arrives anyway, an engine can refuse admission, preempt a sequence, swap its state out, or free it and recompute the prefix later. Every one of those has a latency cost attached. Which is how "no OOM" ends up coexisting perfectly comfortably with cache thrash and terrible tail latency.

Static batching locks a group of requests together for a larger span of work. Padding makes the short prompts pay for the longest prompt's shape, and a short generation leaves its slot idle until the longer peers catch up. New arrivals wait for the batch boundary. It's simple, and it's fine for homogeneous offline jobs. Interactive traffic almost never arrives or finishes in lockstep.

Continuous batching revisits membership at every iteration boundary. Finished sequences leave, and admitted ones join whenever the token and memory budgets allow. Orca^[5] named the underlying move iteration-level scheduling and paired it with selective batching. Paged KV allocation is what makes changing membership practical, since each request carries a block table instead of a fixed rectangular slab of cache.

The scheduler still has to decide whose latency to spend. First-come, first-served is beautifully predictable right up until a long prompt blocks the line. Prioritize prefills and new-request TTFT improves while ITL stretches for everything already decoding. Prioritize decodes and the streaming cadence holds while the queue ages behind it. Chunking prefills caps the interference. Tenant weights, deadlines, maximum batched tokens, preemption cost, and replica routing all bend the trace further.

The panel's metrics treat request acceptance as time zero. Push the prompt length up to add prefill work, drop the KV budget to force admission pressure, or hold arrivals fixed and compare policies against each other. Its synthetic tick costs are deliberately visible rather than hidden. A production comparison needs real prompt and output distributions, a declared SLO, warm-up, model and tokenizer revisions, kernel versions, accelerator topology, and input/output token accounting kept apart.

Standard attention materializes intermediates, or shuttles them repeatedly between high-bandwidth memory and the faster memory on chip. FlashAttention^[6] tiles the queries, keys, and values, computes attention blocks on chip, and keeps the softmax statistics it needs to combine those tiles exactly. Memory traffic and temporary storage change; dense attention is not approximated. IO-awareness is the whole point — fewer floating-point operations buy you nothing in wall-clock time when data movement is what's actually limiting you.

Quantization changes the representation instead. Lower-bit weights shrink stored bytes and memory traffic. Lower-bit activations can open up different matrix kernels. Lower-bit KV shrinks the per-token cache line. Those are three separate decisions with three sets of calibration, kernel, and quality consequences, and collapsing them into one "we quantized it" hides which one broke. A nominally four-bit weight file can still dequantize back up to higher precision for operations that lack support, carry scales and outliers alongside, or run kernels that underperform at small batch sizes.

SmoothQuant^[7] goes after the activation outliers that make W8A8 post-training quantization hard. For a channel-wise scale s, the transformation divides activation channel X by s and multiplies the matching weight channel by s. Before quantization their product is mathematically identical; what's moved is the difficulty, from activations toward weights, which calibrate more easily under the method's assumptions. The quantized computation can still introduce error of its own.

Quantization stops being free at several different boundaries: when the quality loss crosses whatever the task can tolerate, when conversion or calibration turns operationally fragile, when the target hardware turns out not to have the kernels you assumed, when metadata eats the memory savings you were counting on, or when latency moves the wrong way at your deployed batch shape. Use perplexity only if perplexity predicts your application. Otherwise measure the task evaluation, the prompt languages, structured-output validity, and the tail cases the endpoint has to keep getting right.

Keep the memory claims tied to whichever target you picked. Weight-only quantization cuts the fixed replica line and leaves BF16 KV exactly where it was, so long-context concurrency barely moves once weights have stopped dominating. KV quantization changes bytes/token directly and brings its own accuracy and kernel questions. Activation quantization is mostly aimed at compatible matrix execution and transient storage. You can combine all three in one deployment — just keep three rows in the capacity workbook, so a regression can be traced back to the representation that changed.

A concurrency number without a workload attached is half a sentence. Memory capacity depends on resident tokens. Compute capacity depends on input and output token rates, batch shapes, kernels, and parallelism. Service capacity adds the latency objectives on top, because a request only counts as useful if it met its TTFT, ITL, or end-to-end deadline. What you usually want is goodput — completed work that landed inside the stated SLO.

Build the envelope by sweeping offered load and length buckets rather than quoting a single saturation point. Hold the model and system revision fixed while you do it. Record p50, p95, and p99 for queue delay, TTFT, ITL, and end-to-end latency, plus input tokens per second, output tokens per second, cache occupancy, and preemptions. Then repeat the whole thing for burst shapes that look like your production traffic. Stop turning up the load when the declared SLO breaks, even though aggregate throughput will happily keep climbing past that point.

Use an open-loop load generator when the question is how the system behaves in overload. A closed-loop client that waits for one response before sending the next quietly reduces its own arrival rate as latency rises, which conceals exactly the queue growth you were trying to see. Record offered rate and achieved completion rate as separate numbers. Warm-up should cover model load, graph compilation or capture, allocator stabilization, and cache state — and the measured window should still include the cold behavior real users hit, whenever scale-to-zero or replica churn is part of production.

Replicas add another queueing layer on top of all of it. More replicas cut queue delay and isolate failures, and each one duplicates the weights unless the topology shards the model. Tensor or pipeline parallelism trades that for communication cost and different per-device memory. A router that balances by request count will cheerfully pile long prompts onto one replica while another handles short completions. Token-aware routing and per-replica telemetry are what make that imbalance visible before a customer finds it.

OOM is either a ledger violation or an allocation spike, and the two need different fixes. Break the number apart: load-time weights, runtime peak, logical KV, physical KV blocks, allocator reserve, communication buffers. Record which phase died — model load, prefill admission, decode growth, or graph capture. Lowering maximum concurrency can paper over a per-request cache you underestimated; lowering maximum sequence length can paper over an admission policy with no bound on it.

Queue collapse starts the moment sustained admitted work outruns completion capacity. Queue age rises, TTFT stretches, clients retry, and the retries push offered load higher still. Admission control, bounded queues, load shedding, and retry budgets are what stop a slow service from turning into a positive-feedback loop. Scaling out after saturation only helps if the new replicas come up before the queue and the retry storm have eaten the deadline.

Cache thrash shows up as frequent preemption, eviction, swapping, or prefix recomputation. The engine can sit comfortably below any OOM threshold while useful work falls, because it keeps reprocessing the same prompt tokens over and over. Compare logical live KV bytes against reserved block bytes, allocation failures, prefix-cache hit rate, and recomputed tokens. And test a block-size change against the real sequence distribution — picking it from the average length is how you end up optimizing for a sequence nobody sends.

Where you sit on the stack follows from what you need to own. An engine makes sense when you control the model processes and need direct access to scheduling, kernels, cache policy, and hardware tuning. An endpoint platform makes sense when you want a deployable model service with autoscaling, routing, observability, and operational APIs, while keeping real runtime choices in your hands. Managed inference makes sense when the provider can own the accelerators, the upgrades, the availability, and most of the capacity planning under a contract you're able to test.

Required control, required evidence, and staff time set that boundary. Custom kernels and cache policies mean owning the engine. Data residency, pinned revisions, or strict tail SLOs may mean a dedicated endpoint. Variable traffic and a small operations team point toward managed capacity. Whichever way it goes, keep the same benchmark input record and the same latency vocabulary. Abstraction can move the operations off your plate; it can't make the workload assumptions disappear.

Portability has limits worth naming before you commit. An engine configuration couples model architecture, quantized format, kernel availability, device generation, parallelism, and traffic into one thing. An endpoint contract couples scale behavior, quota, routing, observability, and revision policy. A managed API might expose nothing below request-level metrics. So work out the lowest layer at which your team will have to diagnose a missed SLO. If the cache, scheduler, or kernel evidence you'd need lives below the product boundary, either write that evidence into the contract or own the layer it comes from.