The Model Sheet

A model's name is a label chosen by whoever released it. The specification is a short list of numbers underneath: how many parameters it has, how many of those it uses per token, how many bits each weight is stored in, and how long a sequence it can attend to. Add the fields that record how it was made and modified — what training stage produced it, whether it was distilled from a larger teacher, whether its weights were edited after release — and you can predict whether a model will fit your machine, run fast enough to be useful, and behave the way your task needs.

Most disappointment with local and open models comes from choosing on the name and the leaderboard rank, then discovering the download will not load, crawls at reading speed, or refuses the format you needed. Those are not surprises. They are readable from the spec sheet before you spend the bandwidth.

This drawing reads that spec sheet field by field. It covers how a model is trained and distilled, what abliteration and merging do to released weights, how quantization trades bits for size and speed, and where open weights live and how to vet them. It does the memory arithmetic that decides what runs on a given machine, and it ends with a procedure: define the task, shortlist by specification, quantize to fit, and test on your own work rather than someone else's benchmark.

Take a filename a local runtime actually loads: Qwen2.5-Coder-32B-Instruct-Q5_K_M.gguf. Six fields are packed into it. Qwen2.5 is the family and version. Coder is the domain specialization. 32B is the parameter count. Instruct is the post-training stage. Q5_K_M is the quantization. .gguf is the file format. Every one of those fields changes how the model behaves and whether it runs for you.

The consumer's job is to decompose the name into those fields and check each against a constraint you own: your memory, your latency tolerance, your task, your license terms. A leaderboard collapses all of that into a single rank on tasks that may not resemble yours. Rank is a weak prior. The fields are the decision.

Parameters are the learned weights. More of them raise the ceiling on what a model can represent, at a proportional cost in memory and, for a dense model, in the work done per token. Compute-optimal training work showed that capacity has to be matched to training data rather than scaled alone^[1], which is why a well-trained 32B can beat a poorly-trained 70B. Count is a capacity hint, not a quality guarantee.

The field that matters most for local use is one the marketing name often hides: dense versus mixture-of-experts. A dense model runs every parameter for every token. A mixture-of-experts model routes each token to a small subset of expert sub-networks, so it stores a huge total but activates only a fraction per token^[2]. The specification then carries two numbers — total parameters and active parameters — and they do different jobs. Total decides whether the weights fit in memory. Active decides how fast tokens come out.

Mixtral made the pattern ordinary in open weights: eight experts, two active per token, near the quality of a much larger dense model at a fraction of the per-token compute^[3]. For anyone running locally, this is the single most useful thing to understand. A 100B-total, 12B-active MoE gives you close-to-frontier quality at the speed of a 12B model, provided the full 100B fits in memory. Sheet 7 turns that into the arithmetic.

The specifics of the routing vary and are worth a glance in the model card. Newer designs use many more, smaller experts — dozens or hundreds — and route to several per token, sometimes with a handful of always-on "shared" experts that every token passes through. The two numbers you care about survive all of that: total parameters, which set the memory footprint, and active parameters per token, which set the speed. When a card lists only total, look for the active count before you judge how fast it will run; a 30B-total MoE with 3B active behaves at the keyboard like a small model wearing a large coat.

A model is built in stages, and the stage it stopped at is the field that most changes how it behaves. Pretraining runs next-token prediction over a large text corpus and produces a base model: it completes text and does not converse, so prompting it like a chatbot produces drift. Instruction tuning continues training on instruction-response pairs, and reinforcement learning from human feedback aligns the model toward preferred answers, producing an instruct or chat model that follows requests and holds a helpful register^[4]. Reasoning training rewards a model for emitting a hidden thinking span before its visible answer, which raises accuracy on hard problems and spends tokens doing it. Two models with the same parameter count can behave nothing alike because they stopped at different stages.

Distillation is a separate lever, and it is worth understanding because it explains why some small models behave far above their size. Distillation transfers what a large teacher model knows into a smaller student^[5]. Instead of training the student only on hard labels — the single correct next token — it trains the student to match the teacher's full output distribution, the soft probabilities the teacher spreads across the whole vocabulary. That distribution carries much more information per example than a one-hot target: it records that the teacher thought the answer was probably A, plausibly B, and never C. The student learns the teacher's relative preferences, not only its top choice.

Because it inherits structure the teacher already discovered, a distilled model usually outperforms a same-size model trained from scratch. DistilBERT reproduced most of BERT's performance with roughly 40% fewer parameters and ran faster for it^[6]. In consumer terms, a 7B or 8B "distill" of a much larger model can behave closer to a mid-size model than its parameter count suggests — you get some of the teacher's competence at the student's memory footprint and speed. That is a strong bargain on a laptop.

The pattern now shows up most often in reasoning models, and it is written into the name. A large reasoning teacher generates long chains of thought, and a smaller student is fine-tuned on those traces until it reasons in the same style. On Hugging Face a tag like DeepSeek-R1-Distill-Qwen-7B reads as a Qwen-7B student taught to reason by a DeepSeek-R1 teacher: the architecture is Qwen, the behavior came from R1. Read a Distill field as student-then-teacher, and expect the student to carry the teacher's habits at the student's ceiling.

Distillation is also why the small models you can run keep getting better without your machine changing. Frontier labs train an enormous flagship, then distill it into the compact sizes people actually deploy, so each generation of an 8B model starts from a stronger teacher than the last. The alignment steps and the reasoning steps described above can happen on the teacher, the student, or both — the model card is where a serious release records which. When a card says a small model was distilled from a named larger one, that lineage is a better predictor of behavior than the parameter count alone, and it is the kind of field a leaderboard rank throws away.

Beyond the official training stages, the community modifies released weights, and those modifications carry their own tags. A fine-tune continues training on a narrower distribution to specialize a base or instruct model toward a domain, a format, or a persona. A merge blends the weights of two or more models — averaging or interpolating their tensors — to combine behaviors without any training at all. An abliterated model has had its refusal behavior removed by editing activations rather than by retraining. Each tag means a different thing was done to the weights.

Abliteration is the one worth understanding in mechanism, because it explains what the "uncensored" downloads actually are. Interpretability work found that in many chat models, refusal is largely mediated by a single direction in activation space: one linear feature whose presence pushes the model toward declining a request^[7]. You can estimate that direction from the difference between the model's activations on harmful and harmless prompts, then subtract it — orthogonalize the weights against it at every layer — and the model stops refusing, with no gradient training involved. The name is a blend of ablate and obliterate: the refusal direction is removed.

Merging deserves its own note because it is common and invisible in the name. A merge combines the tensors of several models with no training: spherical interpolation blends two models weight-for-weight, and task arithmetic adds the difference a fine-tune introduced onto a different base, so a "coding" adjustment can be grafted onto a "chat" model. Merges can produce a capable model cheaply, and they can also produce a confident one that quietly lost a skill neither parent's card would predict. The only defense is the same one abliteration needs: read the card for what was combined, and test the result.

Removing a direction is not free either. The same direction can carry signal the model relied on elsewhere, so abliteration can degrade calibration, weaken instruction-following, and remove refusals of genuinely bad requests you did want kept. Provenance compounds the risk: an abliterated merge inherits whatever both parents did, and the lineage can be hard to reconstruct from the filename alone. The org huihui-ai publishes many abliterated variants; they are useful for research into model behavior, for red-teaming, and for tasks where a well-meaning model over-refuses legitimate content — and a poor foundation for a production feature that needs stable, calibrated behavior. Editing a model's safety behavior is a choice you own the consequences of, so match the modification to a real need and keep the plain instruct model as your default.

A trained weight is a number, stored by default in 16-bit floating point (FP16 or BF16). Quantization stores it in fewer bits — 8, 6, 5, 4, sometimes 3 or 2 — together with a small amount of per-block scaling metadata: a shared scale, and often an offset, that maps the low-bit integers back to an approximate range. Fewer bits mean a smaller file, less memory traffic per token, and faster generation, paid for in precision. The research line runs from 8-bit inference that held accuracy at scale^[8], through post-training methods that quantize to 4 bits with careful error correction^[9], to 4-bit representations designed so a model can still be trained through them^[10].

In the GGUF files most local runtimes load, you will see labels like Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, Q2_K. The number is the approximate bits per weight. The _0 suffix is a legacy scheme that quantizes every weight in a block the same way. The _K suffix marks the k-quant family, which allocates bits unevenly across a block — spending more on the weights that matter and fewer on the rest — and holds more quality at the same average bit rate. The trailing _S, _M, or _L is the size variant within a k-quant level, trading a little more file size for a little more fidelity. So Q4_K_M reads as a medium 4-bit k-quant, which is the usual default.

A further refinement uses an importance matrix, or imatrix. The quantizer runs a calibration text through the model, measures which weights most affect the output, and protects those during quantization. Files built this way are often tagged i1 or "imatrix," and the very low-bit i-quants — IQ2, IQ3 — lean on the technique heavily, holding usable quality at bit rates where a plain quant collapses. At the same bit rate an imatrix quant generally beats a non-imatrix one; the gap is large below 4 bits and small at Q5 and above.

Weights are the large cost, but they are not the only thing you can quantize. The key-value cache also grows with context, and most runtimes can store it at 8 or even 4 bits, trading a little quality for a much longer context in the same memory — a useful lever when a model fits but its context does not. And a subtlety behind all of this: quantizing weights is comparatively safe because a shared scale recovers most of their range, while quantizing the activations that flow through the model is harder, because a few outlier activations dominate and clip badly. That difference is what the research above navigates, and it is why 8-bit weights held accuracy long before 4-bit did^[8].

One more split matters when you choose where to run. Three quantization families dominate, divided by hardware. GPTQ and AWQ are 4-bit schemes aimed at GPU serving stacks like vLLM and TGI: they keep weights in a form fast GPU kernels consume, and AWQ specifically protects the weights that activations lean on most. GGUF is the format for CPU-and-Metal runtimes — llama.cpp, LM Studio, Ollama — and it carries the full ladder from Q8 down to Q2. The practical rule is short: GGUF for a laptop, GPTQ or AWQ for a GPU box.

The ladder also settles a recurring question: given a fixed memory budget, do you run a bigger model at a lower quant or a smaller model at a higher one? For models in the same family, the bigger model at Q4 usually wins over the smaller model at Q8, because parameters buy more capability than the last bits of precision preserve — until you drop past Q3, where the bigger model's damage overtakes the smaller model's headroom. The demo makes the trade concrete: set a memory limit, then watch which rung of a large model still fits and what it costs in quality against a small model sitting comfortably at Q6.

Open weights are distributed as repositories, most of them on the Hugging Face Hub. A repo lives under an org or user — the namespace before the slash in meta-llama/Llama-3.3-70B-Instruct — and that namespace is your first provenance signal: the original lab, or a re-packager working from it. Inside, the repo carries the weights, a config describing the architecture, a tokenizer, a license, and a model card, which is the README. Reading the card is the step most people skip and most regret skipping.

Read the model card for what the filename omits: the license, the base model this one derives from, how any merge or abliteration was produced, the intended use, and often prompt-format notes and benchmark numbers. A capable model with an unusable license or an unexplained lineage is a poor foundation regardless of how it scores.

Two file formats dominate, split by where they run. Safetensors holds weights at full or near-full precision in a safe, memory-mappable container; it is what GPU serving stacks and fine-tuning load, and what the original lab usually publishes. The "safe" in the name is literal: it replaced older Python pickle checkpoints, which could execute arbitrary code on load, so a safetensors file carries data only and cannot run anything when you open it. GGUF is the quantized single-file format that local CPU-and-Metal runtimes load, and it is what most people run at home. The original repo is usually safetensors; the GGUF you actually download is usually published by a separate quantization provider who converted it.

A few repo details trip up first-timers. Large weight files are split into numbered shards named like ...-00001-of-00003.gguf; the runtime reassembles them from the first, so download the whole set rather than the one file whose name you recognize. A vision-capable model ships an extra mmproj file — the multimodal projector that maps image features into the language model — which you only need, and must load alongside the weights, if you want image input. Some repos are gated: you request access, agree to the terms, and the org or an automated gate grants it before the download unlocks.

Licenses vary more than the phrase "open weights" suggests. Some open-weight models are permissive; others restrict commercial use, set monthly-active-user thresholds above which you need a separate agreement, or forbid training other models on their outputs. Open weights is not the same as open source. Read the license as a constraint you own, the same way you read the memory number.

Two numbers decide whether a model is usable on a given machine, and both are readable before you download. The first is whether it fits. Weight memory is close to parameters times bits-per-weight divided by eight, plus a key-value cache that grows with context length and the number of concurrent requests. The sum has to sit inside usable memory — for a GPU, its VRAM; for Apple Silicon, the share of unified memory the system will wire for the GPU.

The second number is speed. Token generation is bound by memory bandwidth, because each token streams the active weights out of memory. Divide the machine's bandwidth by the bytes read per token and you get an order-of-magnitude tokens-per-second, before software overhead. This is why a mixture-of-experts model is the sweet spot on a big-memory machine: total parameters set the fit, but only the active parameters are streamed per token, so a 100B-total model with 12B active reads like a 12B model and generates several times faster than a 70B dense one that codes no better.

Two speeds hide inside that one number, and they fail differently. Prefill reads your whole prompt at once and is compute-bound, so a long prompt or a big document adds a pause before the first token appears. Decode produces the reply one token at a time and is the bandwidth-bound rate above. A model can start slowly on a long prompt yet stream quickly once it begins, or the reverse — worth knowing when a run feels sluggish, because the fix differs. Concurrency changes the arithmetic again: serving several requests at once shares the weight reads across them and raises total throughput, at the cost of a larger key-value cache and more memory. On a single-user laptop, ignore concurrency and read the decode rate; on a shared host, it is the lever that pays for a GPU.

A public benchmark measures a task that someone else defined, and the popular suites leak into training corpora over time, so a high score can reflect exposure as much as ability. A model that tops a coding leaderboard may still fail your codebase's conventions, your prompt format, or your latency budget. The rank narrows the field. Your own test decides.

The test does not need a framework. Collect ten to thirty prompts that represent the work you actually do, each with a known-good or clearly-judgeable answer. Run every candidate through the same set. Score each output pass or fail against what you needed, and record two operational numbers alongside: tokens per second and, for hosted models, cost per run. A model that passes nine of ten of your prompts at usable speed is a better choice than one that ranks higher on a suite you will never run.

Pass or fail beats a numeric score because it forces you to write down what "good" means before you look at the output, which is where honesty lives. Include the cases that break models: your exact output format, a long input near your real context length, an instruction the model should refuse or push back on, and one task from the hard end of your work rather than the easy end. Keep the prompts and the expected answers in a plain file under version control so the set is a fixed target and not a memory. When a tempting new model appears, the test is already written, and the only question is whether it earns the swap.

Reading the fields turns into a repeatable procedure. Define the task and its constraints first: the format you need, the latency you will tolerate, the privacy and license terms, the budget. Shortlist by specification — filter to models whose size and post-training stage match those constraints before you look at any rank, and read the tags for distillation, merges, or abliteration so you know what each candidate is. Quantize to fit — pick the quantization that lands inside your memory with room for context, using the fit equation rather than the largest download. Test on your own eval — run the shortlist through your prompt set and score it the same way. Deploy the one that passes, and keep the eval so the next release has to earn the swap.

Keep the eval after you choose, because the decision is not permanent. Models are released weekly, quantizers re-quantize, and your own tasks drift; the prompt set is what lets you re-run the comparison in an afternoon instead of re-forming an opinion from scratch. Re-test when a model in your family gets a new version, when a task starts failing in production, or when a smaller model appears that might free up memory for a longer context. The eval turns "should I switch?" from a gut call into a measurement.

The last field is the boundary between local and hosted. A big-memory local machine has a real ceiling: the best model you can run is the largest capable one whose weights fit, which for a mixture-of-experts design can be near-frontier and fast, while the very largest open models stay out of reach at usable quantization and belong on a GPU host or an API. Local buys privacy, zero marginal cost, and offline use; hosted buys the frontier models and elastic concurrency. Most serious setups end up with both, and the model sheet is how you decide which job goes where. For the memory-and-bandwidth side of that ceiling, The Transformer covers how the weights and cache are used per token; for fitting evidence into a bounded prompt once a model is chosen, The RAG Pipeline covers the context budget; and My 2026 AI Engineer Setup shows where these choices land in daily practice.