The Benchmark

A benchmark score is a manufactured number. Someone picked the tasks. Someone else sampled the items, settled on a prompt format, pinned a model version, ran it, scored the output, decided what to do about the runs that failed, and averaged whatever survived. Every one of those steps is a choice, and every choice leaves a trace. Strip the traces away and a leaderboard row can't tell a buyer whether the model will answer their support tickets, repair their repository, summarize a clinical note, or come in under their latency budget.

This drawing runs three instruments through that same pipeline. MMLU asks fixed-answer multiple-choice questions across academic subjects^[1]. SWE-bench hands a system a real repository and a real issue, then lets the repo's own tests judge the patch^[2]. Chatbot Arena shows two anonymous answers to a person and asks which one is better^[3]. All three publish a number, and the numbers answer different questions, because the samples differ, the interfaces differ, the scorers differ, and the quantity each one is actually estimating differs.

So the question worth asking about any score is narrow: does this evidence reduce my uncertainty about the decision in front of me? An evaluation earns its keep when it connects the use you have in mind to tasks and consequences that resemble it. Ranking is a convenience that arrives afterward, and whether it deserves any weight comes down to provenance, uncertainty, the slices underneath the average, contamination checks, and operating cost.

Start at the right edge of the drawing, with a decision. A technical PM picking a model for a support workflow. An engineering leader deciding whether to fund an agent roadmap. A journalist trying to check a vendor's claim. Work backward from that choice to the capability and the constraints that actually matter, and only then go looking for evidence that represents them. Run it the other way — open a leaderboard first — and the benchmark quietly gets to define your problem for you.

The pipeline starts with a target population, something like "production bug reports in maintained Python services." A sampling frame makes that population reachable, and the rules you write into it produce the items: which repositories count, which languages, which time window, how hard a case has to be. A prompt adapter turns each item into model input. The run records what version answered, with which tools, at what temperature and token budget, on what date. The scorer turns outputs into item results, aggregation turns those into a headline and an interval, and the report maps all of it back to the decision.

Every arrow on that drawing can move the final number while the model weights sit perfectly still. Reword the instruction. Allow a tool. Normalize punctuation before matching. Count a timeout as a zero instead of dropping it. Weight all 57 subjects equally instead of all items equally. Each of those produces a different estimand — a different quantity being estimated — and the score changes accordingly. Provenance is what turns those choices from invisible into inspectable.

Version the whole chain as one evaluation release. A model name on its own isn't enough for an API that changes behind an alias, so record the provider's dated model identifier and keep the raw responses when the terms let you. Hash the item files, the prompt templates, the scorer code, and the environment manifest, then give every run an ID that ties those artifacts to the item-level results and to the invoice. Do that and a score stops being an annotation on a slide and becomes a dataset someone can audit.

Missing items need the same care. An item can vanish because a parser rejected the output, a sandbox died, a request timed out, a safety system refused, or a human looked at it and threw it out. Each cause tells you something about how the system behaves in deployment. Write the policy down before the run, report the counts by cause, and keep failures in the denominator unless the estimand you declared genuinely excludes them. A selective denominator can push a score up while the completion rate you actually care about falls.

A benchmark sample stands in for a larger population, and the substitution is where most of the trouble starts. MMLU assembled 15,908 multiple-choice questions across 57 subjects, from elementary mathematics up through professional law. The subject inventory buys real breadth. What it doesn't buy is escape from the format: these are written exam questions with four supplied choices, and the score measures performance under that interface, not open-ended work across every domain the subject names evoke.

Sampling error starts long before anyone computes a confidence interval. A convenient collection tends to leave out the hardest customers, the private codebases, the regulations that changed last quarter, the uncommon languages, the very long contexts, and the cases where the right move is to decline to answer. Deduplication can strip near-copies and still leave whole template families over-represented. A stratified sample can guarantee you enough high-cost or low-frequency cases to estimate them at all, and then you reweight to production prevalence when you compute the deployment estimate.

Prompt format is part of the instrument, not a detail around it. Hendrycks and colleagues reported five-shot GPT-3^[4] accuracy of 43.9% on MMLU in the benchmark's September 2020 paper, and that number is welded to that model, that prompt regime, that item set, and that date. Change the instructions, the answer-extraction rule, the order of the choices, the examples, or the chain-of-thought policy, and the measured score moves while the checkpoint underneath it doesn't. Any report worth reading publishes the rendered prompts, or a deterministic adapter that recreates them.

Prompt sensitivity is diagnostic rather than embarrassing. If harmless formatting moves a score far enough to flip a purchase, that variation is part of the system and belongs in the measurement. Production ships templates, parsers, retrieval, and tool policies along with the weights. Evaluating the naked model can answer a research question while missing the unit you're actually going to deploy.

Representative sampling depends on time, too. Product traffic moves, code dependencies move, regulations move, and the words users type move with them. A random sample from last year can be statistically precise about a population that no longer exists. Stamp the collection window, watch how task and outcome frequencies drift, and schedule refreshes. Keep the old sets around for longitudinal comparison, but don't let continuity outrank current validity: a score that holds steady on stale work sits perfectly comfortably next to production utility that's falling.

The three instruments share the pipeline and part company at the task contract. MMLU shows a question with four choices and scores whether the extracted choice matches the key. That buys cheap scoring, repeatability, and broad subject labels. It also brings guessing, answer-key errors, prompt sensitivity, and a weak connection to open-ended work.

SWE-bench starts from a GitHub issue and a repository snapshot. The system has to produce a patch, and the repository's tests decide whether it resolved anything. The original October 2023 paper reported that its best evaluated model resolved 1.96% of tasks — a dated result describing that paper's harness, systems, repository versions, and task set. Later leaderboard numbers run different agents with different tools and budgets against benchmark variants, so they need their own provenance rather than inheriting the paper's. Executability makes the oracle much stronger. Flaky tests, broken environments, underspecified issues, and patch leakage are the risks that come with it.

Chatbot Arena shows one user prompt to two anonymous systems and asks a person which response is better, or whether they tie. The March 2024 paper analyzed more than 240,000 votes and fit a Bradley–Terry model^[5] to rank the systems. Preference picks up usefulness and style that exact matching can't see. It also folds in the user population, the prompt distribution, position effects, response length, which models happened to be available, and traffic that keeps changing. What the score estimates is comparative preference inside that arena, not factual accuracy in isolation.

No conversion factor makes these three scores interchangeable. A fixed key observes selection under an exam format. Tests observe whether a code change satisfies conditions someone wrote down. Votes observe which of two displayed responses a person preferred. Pick the instrument whose evidence resembles the consequential part of your decision, and treat the other two as context.

Scorers deserve their own evaluation set, which almost nobody builds. Seed it with outputs you know are correct, partially correct, harmful, and irrelevant, plus adversarial formatting and plausible-sounding falsehoods. Then measure how often the scorer agrees with a qualified human, how flaky the tests are, which way a model judge leans, and how calibration varies by slice. Freeze the scorer version for the duration of a comparison. When a judge or a test suite changes, that's a new series — label it as one instead of joining incompatible scores on a single trend line.

An item scorer produces data, not a verdict: exact correctness, tests passed, a rubric vector, a preference, latency, cost, or several of those at once. Aggregation is where you decide what counts equally. A micro-average weights every item the same. A macro-average can weight each subject or each customer the same. Utility weighting values outcomes by how often they happen or how much they cost when they go wrong. These are policy choices wearing arithmetic.

Take two systems across knowledge, code, and service tasks. In an illustrative fixed dataset, Model A scores 92, 46, and 40; Model B scores 78, 76, and 72. Weight the three slices equally and B wins. Weight by a portfolio dominated by knowledge questions and A wins. Neither calculation is fraudulent, and neither is wrong — they estimate different target mixtures. Publishing only the winning mean is what hides the assumption that picked the winner.

Means also bury tail failures. A support model can average 90% while failing nearly every cancellation request. A coding agent can sail through routine dependency bumps and corrupt migrations. So the aggregate belongs next to slice counts, intervals, severity, abstentions, timeouts, and worked examples^[6]. A small severe slice can justify blocking a release even when its population-weighted contribution barely nudges the mean.

Simpson's paradox is the sharper version of the same warning. A candidate can do better inside every meaningful slice and still look worse overall, if the two candidates were evaluated on different slice mixtures. Paired evaluation on identical items kills that particular error outright. When traffic allocation genuinely differs, standardize both systems to one declared population before you rank them. And publish the count and the weight next to every slice score, because a percentage without a denominator hides how much evidence is behind the bar.

The demo holds prepared item-level values fixed. Exact match, unit-test, pairwise-preference, and rubric-judge modes impose different score mappings. Task weights change the target population. Judge noise moves subjective modes. Sample size changes interval width. The contamination toggle grants one candidate an artificial advantage. The ranking reorders because measurement choices have changed, not because the model learned during the interaction.

For RAG stage metrics, retrieval recall and answer faithfulness belong on separate lines before any composite. The existing RAG evaluation guide applies that separation to a specific system. The rule generalizes: keep the diagnostic measures visible underneath the decision metric.

A score computed on a sample is an estimate, and estimates come with error bars whether or not anyone prints them. If binary item outcomes were independent draws from one population, the rough standard error would be √[p(1−p)/n], and a 95% normal interval runs about 1.96 standard errors either side. Near 50%, 100 independent items give you a margin of roughly ten percentage points. A thousand items gets you to about three. Small benchmark subsets simply cannot support fine-grained rank claims, however confidently the row is printed.

Real evaluations break those assumptions immediately. Items share templates and sources. A dozen questions come from one repository. Repeated samples from the same model share a prompt. Human votes cluster by user and by time of day. Model judges carry systematic preferences. The fix is to bootstrap by the true sampling unit, cluster the observations that travel together, repeat the stochastic runs, and say out loud which sources of variation your interval covers and which it doesn't.

Pairwise systems add a dependency of their own: a rating is relative to the opponents and the traffic the system actually saw. Elo is the familiar version; Bradley–Terry estimates latent strengths from win probabilities. Both depend on who played whom, so a model can gain or lose rank because the field changed or the prompt mix drifted, without anything about the model changing at all. Publish the vote counts, the pairing policy, the interval method, and the snapshot date. "Rank 3" on its own advertises a precision the experiment may not contain.

Practical significance comes after statistical uncertainty, not instead of it. A one-point quality difference can be irrelevant next to a fivefold cost difference, or decisive in a high-volume safety slice. Decide on the smallest decision-relevant difference before you look at results. If the intervals straddle that threshold, either go collect more representative data or accept that the evidence doesn't distinguish the candidates.

Multiple comparisons make leaderboards especially good at producing lucky winners. Throw dozens of models, prompts, and checkpoints at one set and the highest observed score has selection noise baked into it. Report the search process, keep a final untouched set, and soften the claim in proportion to how many hypotheses got tested. A confidence interval computed after you've chosen the winner doesn't contain the uncertainty that choice created. Replicating on fresh items is worth more than another decimal place on the development set.

Uncertainty should also cover operational randomness when the decision includes it. Retry policies, temperature, retrieval updates, tool failures, and human review queues all move outcomes. Run enough repeats to estimate within-item variance, then keep it separate from between-item variance. Paired differences usually need fewer items than independent comparisons, because both candidates face the same tasks. Archive seeds where the provider supports them, remembering that hosted systems can stay nondeterministic under byte-identical requests.

Contamination is what happens when evaluation information reaches the model's training, selection, prompting, or tool setup in a way that won't be available in the target use. Direct train/test leakage is the clean case: benchmark questions or their solutions land in pretraining or fine-tuning data^[7]. The less visible routes are near-duplicates, translated copies, answer explanations posted online, merged repository patches, and benchmark-specific tutorials. Even with raw items kept out of training entirely, tuning repeatedly against a public leaderboard leaks test feedback into development.

How much it matters depends on the instrument. A fixed-answer item can be memorized verbatim. An executable repository task can leak through the merged patch or the issue discussion. A preference benchmark drawing fresh live prompts is much harder to memorize, and can still be gamed through style tuning, model identity cues, or sustained optimization against arena users. Private tests and held-out prompts slow adaptation down. They don't establish permanent purity, and nothing does.

A contamination register is the artifact that makes any of this tractable: item source, public date, suspected exposure route, similarity evidence, affected model versions, risk, and what you did about it. Deduplicate with exact hashes and with semantic similarity, look hard at suspiciously distinctive strings, use canaries where it's lawful to, compare performance by item age, and keep a post-cutoff set in reserve. None of that proves absence. Training corpora are usually undisclosed, and semantic knowledge transfers legitimately all the time.

Read a suspected lift by triangulating. Does it concentrate on old public items? Does performance collapse on paraphrases, or on freshly authored analogues? Does a clean temporal holdout preserve the gain? Put uncertainty around the diagnosis either way: "contamination detected" needs a definition of exposure and evidence to match, and "no contamination" almost always claims more than the audit can support.

Contamination and legitimate learning can look identical from the outside. Reading public documentation before solving a task is expected in most jobs; memorizing the private acceptance answer is not. So define what the deployed system would be allowed to see, and judge exposure against that boundary rather than a universal ban on prior knowledge. In an open-book workflow, web access can improve validity. In a closed-book knowledge probe, the same access destroys the construct. The rule follows the decision.

Temporal splits help when the training and publication dates are both credible. Put post-cutoff issues or questions in a clean partition and compare them against older matched items, controlling for topic and difficulty so recency doesn't become the explanation for everything. For repository tasks, keep the issue, base commit, environment, tests, and patch timestamps. A date attached only to the leaderboard snapshot can't establish that the test material postdated training.

Goodhart's law describes what happens when a measure becomes a target: the link between the measure and the goal it stood for gets weaker the harder anyone optimizes^[8]. A benchmark rewards whatever reliably raises its score. Sometimes that's genuine capability. Sometimes it's answer-format tricks, the verbosity a judge happens to like, benchmark-specific prompting, test exploitation, selective submission, or spending inference compute no production budget would ever approve.

None of this requires fraud. Teams optimize the published acceptance criterion because that's the rational thing to do with a published acceptance criterion. Leave cost off the leaderboard and a search-heavy agent will buy quality with tokens. Let submitters pick their best run out of many and variance becomes an asset. Let aggregate preference dominate and pleasing style can paper over a factual or safety regression. Keep a benchmark static long enough and development specializes to its item families on its own.

The defenses all work by spreading the pressure out. Hold a final set back and don't touch it during development. Rotate fresh items in. Demand full run logs instead of a cherry-picked attempt. Report cost, latency, tool use, and retries alongside quality. Run multiple measures that fail in different ways, so no single shortcut satisfies all of them. Audit the gains that surprise you. Test transfer on a separate task family. Retire compromised items, and keep the versioned historical results so the record stays readable.

Saturation is ambiguous by nature. A benchmark everyone passes can mean real progress, leakage, item coverage that was too narrow to begin with, or resolution too coarse to separate what's left. Cranking up the difficulty on its own tends to produce puzzles with no connection to any decision. Refresh from the target population instead, and carry the known failure costs across with the items. A hard test is only useful when passing it predicts something you care about.

Write the decision sentence before you pick a benchmark. Something like: "Choose the system that completes these tasks for these users, under these permissions, at this quality, cost, and latency, with these failures treated as release blockers." That one sentence names the population, the unit of analysis, the operating envelope, and the loss function. It also shows you where a public benchmark can contribute evidence and where you're going to have to measure locally.

Build the task inventory out of actual work: sampled logs with privacy controls, tickets, repository changes, analyst requests, and the incidents you already know about. Stratify by how often something happens and by how much it costs when it fails. Freeze an acceptance set and keep it frozen, then run a separate diagnostic set that grows every time you discover a new failure. Be explicit about what unit you're evaluating — a base model, a prompt, a retrieval pipeline, an agent, or the whole service — because those are five different experiments. Record every version.

Choose scorers by what success actually looks like from outside. Exact match fits normalized identifiers and constrained answers. Executable checks fit code and structured operations, when the tests genuinely represent the requirement. Rubrics fit qualities with more than one valid output, provided the criteria, anchors, reviewer calibration, blinding, and disagreement handling are written down. Pairwise preference fits comparative experience and needs factual and policy checks beside it, because preference alone will happily reward an attractive error. Model judges cut the cost and have to be validated against qualified humans on the same distribution first.

Pre-register the aggregation, the exclusions, the sample size, the interval method, and the release thresholds, so the analysis can't drift toward the answer you were hoping for. Evaluate candidates on paired items so task difficulty cancels out of the comparison. Count invalid outputs, crashes, and timeouts instead of quietly dropping them. Report population-weighted utility and the protected high-severity slices side by side. Add per-task cost, p50 and p95 latency, human review minutes, and retry rate. A model that wins on quality outside the operating budget hasn't won the product decision.

Make the release rule capable of returning "no selection." Procurement usually forces a winner even when every candidate misses the minimum quality bar, or when the intervals overlap so badly the ranking is arbitrary. A valid evaluation is allowed to recommend keeping the current system, narrowing the scope, adding human review, gathering more evidence, or postponing the launch. Having that option on the table takes the pressure off the evidence to be more certain than it is.

Keep selection and monitoring apart. The acceptance evaluation decides whether a version may ship; production monitoring watches drift, incidents, cost, latency, and user outcomes after it has; diagnostic tests help engineers localize a failure. Those three can share a task taxonomy and still serve different statistical purposes. Reuse one small benchmark for all of them and you expose it to repeated optimization while making its errors much harder to read.

Run the whole thing as a recurring measurement system rather than a one-time bake-off. New production failures feed diagnostics, and after review they graduate into fresh acceptance sets. Distribution shift triggers resampling; scorer drift triggers recalibration. Public leaderboards stay useful throughout — they screen candidates, expose methods, and supply standardized comparisons. Local evidence is what completes the argument.

The final report should make disagreement productive. A reader ought to be able to challenge whether the sampled tasks represent the use, whether the scorer captures success, whether the weights match the priorities, whether the uncertainty permits a rank at all, or whether exposure compromised the test. Every one of those is a resolvable measurement question. A bare score gives a skeptic nowhere to stand.