Designing AI Evaluations

Imagine that you run an online store and your team has changed its automated support assistant. Before the new version reaches customers, you need to answer a practical question: does it resolve more requests without breaking store policy, becoming too slow, or costing too much? The action you will take—ship the new version, revise it, or keep the current one—is the decision.

An evaluation, usually shortened to eval, is a repeatable test that gathers the evidence needed for that decision. One eval case is one small scenario. A case might give the assistant a customer message—“Cancel order 123 before it ships”—along with the relevant order record and cancellation policy. It would describe what a good response must do, what it must never do, and what information should be saved so a reviewer can check the result.

The thing being tested is the system: the model, the learned engine that predicts a response from its input, plus its prompts, which are the text instructions sent to it. The system also includes its data, code, and tools, outside operations it can request, such as looking up or cancelling an order. A task is a kind of work the system should perform, such as checking an order or explaining a return policy. The protocol is the exact recipe for running every case. The saved inputs, outputs, and step-by-step records are the evidence. A scorer is a rule or reviewer that decides whether that evidence shows success. The release rule says which results lead to shipping, blocking, or further investigation. A score only becomes useful after these pieces are clear.

This method works for any artificial-intelligence (AI) application, meaning software whose behavior depends partly on a model. A classifier uses a model to choose a category, such as “return” or “cancellation.” A retrieval system finds source material and supplies it as context, the information available to the model while it answers. A workflow follows a designed sequence of steps and service calls. An agent is a model-guided program that can choose its next step or tool. An agent may change state, the current condition of data or the outside world, inside an environment made from the tools, services, and information it can access.

Those arrangements are different architectures, but they share one evaluation loop. Specify the claim, the statement the result should support. Sample a manageable set of cases from the larger body of real work. Run the system, save the evidence, score it, inspect failures, make the decision, and add newly discovered failures to the next test set.

The Benchmark explains a benchmark: the same standardized test, given under the same rules, so several systems can be compared on a shared set of tasks. This drawing covers the local counterpart, a product eval designed around one product, its users, and the failures that would change a release decision. Public benchmarks can suggest candidates, meaning versions worth considering. Product evals decide whether a particular candidate is ready for a particular job.

Keep four questions in mind while reading the eight sheets:

1. What decision must this eval support?
2. What exact system version will run?
3. Which cases represent ordinary work and rare, costly failures?
4. What evidence and scoring rule will justify the final action?

The first line of an eval should name the decision it supports. “Candidate B feels better” gives another evaluator nothing they can reproduce. A useful version is: “Ship candidate B if, on support requests that resemble real work, it gives more answers supported by the store’s records and policies, does not get worse on any high-risk group of cases, and stays within the response-time and cost limits.”

Evaluation has short names for each part of that sentence. A grounded answer is supported by the sources supplied to the system. A representative request resembles the work real users send. A slice is a group of cases reported separately; a policy-sensitive slice groups cases where breaking a rule would matter. A regression means the new version got worse. Response time is latency. The population is the full set of users, requests, languages, and conditions to which the result should apply. Correctness, safety, speed, and cost are separate quality dimensions. Their acceptable limits form the operating envelope.

That sentence can be expanded into six fields:

1. Decision — what will happen after the test: ship candidate B, keep the current version, roll back, or investigate an uncertain result.
2. System — the exact version of the model and every component around it.
3. Population — the real-world work the cases are meant to represent.
4. Protocol — how every case is prepared and run, including tool access, time and cost budgets, and repetitions. It also records sampling parameters, the settings that control how a model chooses among possible responses.
5. Scorers — how each result will be judged. Deterministic checks return the same verdict from the same evidence. Reference comparisons compare the result with a known-good answer. Model graders ask another AI model to apply a rubric, a written checklist of observable criteria. Human reviewers handle judgments that require context.
6. Release rule — the thresholds, or minimum acceptable results, and the failures that block release.

For the cancellation case, these fields produce a test anyone can follow. The input is the customer message, order record, and policy. The system may read the order and request a cancellation through an approved tool. The evidence includes the sources it read, the tool request it made, the tool result, and its final reply. The case passes only if the order is eligible, the tool changes the correct order, and every claim in the reply agrees with the saved records.

The eval harness is the code and data machinery that runs this contract repeatedly. It sends each case through the versioned system, the exact recorded version of every relevant part, and retains a trace, a step-by-step record of what happened. It then reports results for the whole sample and for each named slice. Failure cost describes how much harm a wrong result could cause, from a minor inconvenience to a safety or privacy incident.

Read Figure 1 from left to right. The decision determines which tasks and users matter. The harness runs those cases through one recorded system version and saves the evidence. Scorers judge the evidence. The bottom box applies the prewritten release rule, and confirmed failures return as future test cases.

The contract prevents a category error, which happens when one kind of measurement is used to support a different kind of claim. A high answer-quality score cannot establish acceptable latency. A retrieval metric, such as the share of relevant documents successfully found, cannot establish that the final answer is factually correct. An offline test run on saved cases before release cannot establish that production traffic, the requests arriving from real users, still matches the sampled population. Each claim needs evidence produced under conditions that resemble the use it is meant to support. The result also needs an uncertainty estimate, a range showing how much the measured value might change with another sample or run. NIST places the same requirement on test, evaluation, verification—checking that the system was built to its specification—and validation—checking that it serves its intended use: document the methods and uncertainty, test before deployment, when a version is placed into use, and continue measuring in operation^[2].

An eval result belongs to a complete system version, an unchanging label for the exact setup tested. Recording only the model name leaves too much unknown. The same model can behave differently after a prompt edit, a document update, a code change, or a tool failure.

The system manifest is the inventory needed to recreate that setup. Start with the model identifier, its exact name and revision. Save the system and developer prompts, the application’s higher-priority instructions to the model. Save the retrieval index used to search documents or a corpus snapshot, a frozen copy of the source documents. Save each tool definition, which names an operation the model may request and the inputs that operation accepts.

The manifest also records application code, sampling settings, and the context limit, the maximum amount of input the model can read at once. A timeout stops a step after a set wait. A retry policy says which failed steps may be attempted again. The runtime environment records the hardware, software, and connected services used for the run.

These details explain changes that a model name cannot. A new retriever, the component that finds source material, may improve factuality, or the correctness of factual claims, while the generator that writes the answer stays fixed. A shorter context limit may cut off a crucial policy paragraph. A tool timeout may turn a capable workflow into a failed task.

Next decide exactly what will be judged. Evaluators call this the unit of analysis. For a routing classifier, it may be one label that chooses the system’s next path. For a retrieval-augmented generation system, usually shortened to RAG, there are at least two units: the documents it found and the answer it wrote. The found documents are the retrieved set. The model’s answer is also called a completion.

A RAG system carries its own evaluation surface: whether the retriever found the right sources, whether those sources were relevant to the query, and whether the answer stayed faithful to them. My Pinecone guide to RAG evaluation develops those retrieval and generation metrics in more detail.

Other systems leave different things to inspect. A document extractor may produce a structured object, such as a JavaScript Object Notation (JSON) record with named fields. A coding system produces a patch, a set of code edits that can be run and tested. A tool-using system produces a trajectory, the ordered sequence of decisions and actions it took, and a final state, the condition of the outside data or environment after those actions. These saved outputs are artifacts. One eval may keep and score several artifacts separately. The complete collection of settings that produced them is the configuration.

Figure 2 has three levels. The top bar is the system manifest. The middle row shows several possible artifacts. The bottom bar is the shared evidence record that connects an artifact to the exact run that produced it.

The evidence record contains more than the final answer. Store the rendered prompt, the exact text after variables such as the customer’s order number have been filled in. Store the retrieved documents and their ranks, or positions in the returned order. Store tool arguments, the inputs sent to each tool, and the results returned. Record any parser error, which means software could not convert an output into the expected structure.

Also store token counts—tokens are the small chunks of text a model reads and writes—along with latency, cost, and final state. Some systems support a random seed, a saved starting value used to repeat randomized choices. Give every run a unique run ID so its records can be connected later.

This trace separates two very different problems. A capability failure means the system could not perform the task. A plumbing failure means a surrounding connection or data flow broke. A “wrong answer” might come from an outdated document, a policy paragraph cut off by the context limit, a tool request in the wrong format, an expired access credential, or a scorer bug. Keeping the evidence makes the result possible to audit, meaning inspect later against a recorded history.

OpenAI’s evaluation guidance treats a single-turn call—one request followed by one response—workflows, and agents as different architectures with different points of nondeterminism, the ability to produce different outcomes from seemingly identical runs^[1]. The practical consequence is local instrumentation, logging added around each component so its behavior can be observed. Evaluate components in isolation, one at a time, when a diagnosis requires it, then run the assembled system to measure the behavior users receive.

An eval dataset is the collection of cases used for the test, and it makes a claim about a population. If support traffic is sixty percent order status, twenty percent returns, ten percent cancellations, and ten percent everything else, a test set made entirely of polished frequently asked questions measures a different product. Representative cases estimate everyday performance. Edge cases cover unusual but plausible conditions. Adversarial cases deliberately try to expose a weakness, such as a request crafted to bypass a safety rule. Both can reveal whether a rare failure crosses a release boundary, a condition that blocks shipping. They belong in the suite, the full collection of eval cases, and their results should remain visible as separate slices.

Begin with real work. Production logs, records created while the live application serves users, reveal actual phrasing, missing context, and unexpected combinations. They also reveal the long tail: many kinds of requests that each occur rarely. Historical incidents preserve failures that already caused trouble. Domain experts, people who understand the subject or policy deeply, can write cases for important rules and supply known-good answers.

Use generated cases to fill specific holes. Synthetic generation uses software or an AI model to create cases. A coverage gap is an important behavior the current set does not test. Before a generated case becomes evidence, a person should check its input, expected answer, and metadata, the descriptive fields attached to it. OpenAI recommends combining real production cases, historical failures, expert-written cases, and synthetic cases, with typical, edge, and adversarial work represented^[1].

Model-written evaluations can expand a set quickly, but they still require human checking. Their labels are the expected answers or categories. Their behavior distribution is the mix and frequency of behaviors they test. Perez and colleagues checked both when generating 154 behavior datasets^[6].

Teams usually divide the suite into partitions, groups with different jobs. The development set stays visible and small enough to run during each iteration, one cycle of editing and testing. The regression set collects repaired failures and runs whenever a relevant part changes. The holdout stays out of day-to-day editing. It estimates generalization: whether an improvement also works on cases the team has not repeatedly studied.

A separate challenge set can collect adversarial or high-severity cases, meaning failures with serious consequences. Keeping that set separate lets the ordinary sample remain representative while still enforcing strict safety or policy requirements. A zero-tolerance check is the strictest kind: one observed failure blocks the release.

Figure 3 answers two questions at once. The four large boxes sort cases by how often they occur and how costly a failure would be. The three boxes along the bottom show when cases are used: during development, after a bug is repaired, or only for an independent holdout check.

Everyday frequency determines how strongly a case should be represented in the main sample. Failure cost determines how strongly its result should affect the release. Keeping both visible prevents a rare, severe failure from disappearing inside an average while avoiding a test set that misstates its real-world prevalence, or how often it occurs.

Attach metadata to every case: source; collection window, the period when the data was gathered; consent, permission to use the data; handling rules, which govern how it may be stored and shared; slice labels; expected evidence; scorer; and severity, an estimate of the harm if the system fails. Redact sensitive data by removing or masking it before the case reaches a third-party system, a service operated outside your organization. Stamp synthetic cases as synthetic. Retire stale cases when old policy, product behavior, or user traffic makes them misleading. CheckList offers a useful behavioral-testing vocabulary: name the abilities the system should have, then cross each with test types that probe expected, borderline, and stress behavior. That grid helps when an aggregate score, one number combining many cases, hides a specific failure^[4]. The Benchmark covers contamination, accidental exposure of test material during development or training, and sampling error, the difference between a sample result and the true population value. A private eval adds drift, gradual change in the real work, users, or conditions after the test set was collected.

The scorer defines what a passing case means. Choose the most direct instrument, or measurement method, that can prove the claim. If one exact value is required, use exact match. It can check that a router chose the label cancellation. If the output should contain named fields with specific data types, use schema validation. A schema is the formal description of that expected structure.

When the output can be run, test its behavior. Executable tests run code or a database query and check the result. Database assertions check whether a required record or state change exists after a tool acts. When several wordings can express the same correct answer, compare their claims with a reference answer. For qualities such as completeness, tone, and clarity, use a rubric, a written set of observable criteria and score levels. A scorer has validity when it truly measures the quality named in the claim.

The support-assistant case needs several scorers because it can fail in several ways. First check that the response parses, meaning software can read its required structure. Check that the order identifier, such as order number 123, matches the request. Check that every factual claim is supported by the retrieved policy and account record. Check that no prohibited action was attempted, the request was actually resolved, and the tone met a short rubric.

Keeping these measurements separate reveals tradeoffs, where improving one property may worsen another. One combined “quality” score would hide them. HELM, a research project for broad language-model evaluation, calls this a multi-metric approach: report several measurements together. A foundation model is a general-purpose model that can be adapted to many tasks. HELM reported its accuracy, the share of correct results, alongside several other properties: calibration, whether stated confidence matches actual correctness; robustness, whether small or difficult input changes break performance; fairness, whether comparable groups receive comparable treatment; bias, systematic skew in outputs; toxicity, harmful or abusive language; and efficiency, the computing time or resources required^[5].

Figure 4 is a cost ladder. Begin at the lower left with a cheap, repeatable software check. Move upward only when the property being judged requires more context. Human review sits at the top because it can handle subtle meaning, but it costs more and varies across reviewers.

Model graders make open-ended evaluation, where several different responses may be acceptable, more affordable. They also introduce another model into the measurement chain, the sequence from output to recorded judgment. The MT-Bench study compared model graders with people and documented three recurring distortions: position bias, favoring an answer because it appears first or second; verbosity bias, favoring a longer answer; and self-preference bias, favoring text produced by the grader’s own model family, a group of related models built from the same underlying design or training lineage^[9]. Calibrate a model grader by checking and adjusting it against a blinded human-labeled set, where people supplied trusted judgments without seeing which system produced each answer. Give the rubric observable criteria and examples at several score levels. Prefer classification, choosing a named category; pairwise choice, choosing between two answers; or criteria-level scores, judging one property at a time, over an invitation to “judge quality.” Reverse answer order in pairwise tests, track disagreement between graders for each slice, and version the grader model and prompt.

Human review also needs calibration. Reviewers can interpret intent and consequences that automated checks, rules run by software, miss, but instructions, expertise, fatigue, and disagreement affect the result. Anthropic’s account of evaluation practice describes the expense and variability, or change from one judgment to another, of open-ended human assessment. It also describes the difficulty of using models to generate or grade tests without inheriting their errors^[3]. Record reviewer instructions, randomize the order of cases and blind reviewers to the system identity where possible, measure agreement, the rate at which reviewers reach the same judgment, and adjudicate ambiguous cases by having an authorized reviewer resolve them instead of silently averaging them.

The distribution of results is the range of possible outcomes and how often each occurs. AI systems are often stochastic, meaning an element of chance can produce different responses to the same request. One passing run proves only that the system succeeded once. Repeated trials, or repeated attempts under the same protocol, estimate how often it succeeds.

Several sources create that variation. Temperature is a setting that controls how freely the model samples among possible next tokens. A provider, the service that runs the model, may update its software. Nondeterministic kernels are low-level computing operations whose exact results can vary slightly between runs. Ties between retrieved documents, the current state of a tool, and previous conversation messages can also change the outcome.

Two familiar quantities answer different questions. A probability is a number from zero to one describing how likely an event is, where zero means impossible and one means certain. In the notation below, k is the number of attempts and p is the probability that one attempt succeeds. pass@k is the probability that at least one of k independent attempts succeeds, where one attempt does not affect another. It is useful when a product may generate several candidates and select one^[7]. pass^k, read “pass to the power of k,” is the probability that all k attempts succeed, useful when every repeated use must work^[8].

Put concrete numbers into the formulas. Suppose one attempt succeeds 60 percent of the time, so p = 0.60, and the system gets five attempts, so k = 5. pass@5 is about 99 percent because at least one of five attempts will usually work. pass^5 is only about 7.8 percent because all five attempts rarely work. The first number describes a product that can generate several options and keep one success. The second describes a product that must work reliably every time it is used.

The formulas assume independent trials with the same success probability. A shared outage, such as one unavailable service, can break many trials together. Repeated retrieval errors, cached state reused from an earlier request, and systematic rubric mistakes that judge many cases the same wrong way create correlated failures, where failures are linked rather than independent. Report empirical results, values measured from actual repeated trials, when the product depends on them. Keep the assumptions beside any analytic curve, a line calculated from a formula rather than measured runs.

Run candidates on the same cases and protocol. A paired comparison gives each candidate the same case, which removes some noise, or irrelevant variation in the result. Report the denominator, the number of eligible cases behind a rate; the point estimate, the single best estimate from the sample; the uncertainty interval, a range of plausible population values; and any exclusions, cases left out and the reasons why. Set the sample size, the number of cases or trials, from the smallest difference that would change the decision. A two-percentage-point product choice—such as 88 percent versus 90 percent—needs more evidence than a twenty-point regression. The Benchmark derives the basic interval logic, the calculation behind uncertainty ranges, and explains why small subsets cannot support precise claims about rank order.

Repeated trials also make the cost ledger, the record of every expense needed for success, honest. The relevant denominator may be accepted answer, resolved request, valid extraction, or completed task. The numerator, the total amount being divided, should include retries, timeouts, retrieval calls, grader calls, and human review.

The top-line score is one summary number for the full test. Use it as an index into the run, then open the results underneath it. Every report should show results by task family, a group of related jobs; language; user group; input shape, such as text length, file type, or presence of an attachment; risk level, the severity of a possible failure; environment; and every slice named in the contract.

Include case counts beside success rates so a strong percentage from a tiny sample is easy to spot. A ninety-five percent average can contain a policy-sensitive slice at sixty percent. A large, easy slice can also hide a regression in work that matters more. The report below shows p95 latency: 95 percent of cases finished within that response time, while the slowest 5 percent took longer.

Read Figure 6 across one row at a time. n is the number of cases in the slice. PASS is the success rate. COST / PASS is the average cost of one successful result. P95 shows the slow end of normal response time. The rightmost column applies the release rule. The overall score is 91 percent, yet the release is blocked because cancellation requests miss their own requirement.

Then read individual failures. For each failed case, place expected evidence, what a correct run should have produced, beside observed evidence, what it actually produced. Inspect the trace, assign a failure category, and identify the earliest component that made the result inevitable. The support assistant’s wrong refund answer might originate in retrieval, policy interpretation, tool arguments, answer synthesis, the step that turns gathered information into the final response, or a grader that accepted an unsupported claim. Fixing the wrong layer can create a passing score while preserving the underlying bug.

Turn confirmed failures into minimal regression cases, the smallest saved examples that will reveal if the same bug returns. Keep the original incident for realism and add a smaller diagnostic test when one component needs isolation. Maintain a failure taxonomy, a consistent set of failure categories, broad enough to guide ownership: source data; retrieval; following instructions; reasoning through a problem; choosing and using tools; formatting the output; applying policy; infrastructure such as networks and hosted services; and scoring. Trends in that taxonomy tell the team where engineering work buys the most quality.

Composite scores, single values made by combining several measurements, can help sort experiments. Release rules should read critical dimensions and slices directly. Weighting assigns more numerical influence to some measurements than others; allowing several excellent style scores to cancel a severe policy violation hides the product decision inside arithmetic.

Continuous evaluation means repeating measurement as the system changes, before and after release. An offline eval provides a controlled comparison on saved cases. Production use reveals new inputs, dependencies—outside components or services the application relies on—and real consequences. Connect the two carefully because an online signal, an observation collected while the live system serves users, is not automatically ground truth, a trusted record of what happened or what the correct result should be.

Run the regression suite on every relevant change. Run the larger acceptance suite, the cases required for a formal release decision, when a candidate approaches release. In production, log the same version and evidence fields used offline by saving them as records. Monitor cost and latency by checking them continuously. Send incidents, failures that affected or could affect users, to an adjudication queue where a person resolves the correct label.

Test a candidate cautiously before full release. In a shadow run, it receives copies of live requests but cannot act on users or production data. A blinded side-by-side comparison shows reviewers two outputs without revealing which system produced either one. Both methods use live-shaped traffic, requests that resemble current use. A canary releases the candidate to a small, limited group first. A staged rollout expands that group in steps. These methods limit exposure while online evidence accumulates.

Read Figure 7 clockwise. A version moves from specification to offline testing, scoring, and staged release. Production observations enter a human review queue. Confirmed failures join the regression set, and a new recent sample checks whether the work has drifted.

User feedback, abandonment, when someone leaves before finishing, escalation, when a case is handed to a person or higher support level, edits, and downstream corrections made later in the process are useful signals. Each can be biased, meaning it reflects some users or outcomes more strongly than others. Users tend to report conspicuous failures, reviewers disagree, and a clicked thumbs-up may reflect politeness or speed rather than correctness. Preserve the raw signal, the observation before interpretation or filtering, sample it for human review, and promote only adjudicated cases into labeled evaluation data, cases paired with trusted expected outcomes.

Keep a rolling holdout, a reserved sample refreshed from a recent production window, to detect drift while repaired incidents enter regression immediately. Do not tune, or repeatedly adjust the system, against the only sample used to estimate current production quality; doing so can make that sample look better without improving new cases. OpenAI’s guidance calls for evaluation on every change, logging during development, and growing the set from observed nondeterminism^[1]. NIST likewise treats measurement as a continuous activity across the system lifecycle, from design through operation and retirement^[2].

The release rule turns measurement into action. Write it before seeing candidate results, because a threshold invented afterward tends to justify the result the team already wants. The baseline is the current system or score used for comparison. The rule states the required improvements, allowed regressions, and critical slices whose failures can block release. It also states the uncertainty policy, which says what to do when the evidence remains inconclusive, along with cost and latency limits and the person who may approve an exception, a documented departure from the rule.

For the support assistant, a rule might require the candidate to improve grounded-resolution success, the share of requests both resolved and supported by the supplied evidence, against the current release. It might preserve performance on order-status and return slices within a declared margin, the largest acceptable decrease; record zero observed prohibited actions in the policy challenge set; stay below ceilings, or maximum allowed latency and cost; and send any cancellation case to human review until that slice clears its own minimum bar. The exact thresholds belong to the product and its risk tolerance, the amount and type of failure its owners are willing to accept. The shape of the rule is reusable.

Record one of four decisions: ship, release the candidate; block, prevent its release; investigate, examine a named problem; or no decision, gather better evidence. “Investigate” means a specific scorer, slice, or infrastructure failure made the result unreliable. “No decision” means the uncertainty interval is too wide or the sample does not represent the claim. Both outcomes protect the integrity of the eval better than forcing every run into a winner.

The downloadable design pack contains an A2 poster, a large 420 × 594 millimeter reference sheet, and an editable worksheet in Portable Document Format (PDF).

The finished run should preserve the contract; the dataset commit, an exact saved revision of the cases; the system manifest; raw evidence; scorer versions; slice report; exclusions; uncertainty; and a signed decision recording who approved the result. That decision packet, the full bundle of materials behind the release, makes a later comparison meaningful. It also shows when a passing release inherited a weak test, when a scorer changed, and when production drift moved beyond the population the eval represented.

The full loop is compact: define the decision; version the system; sample representative, edge, and adversarial work; retain evidence; score with the most direct valid instruments; repeat enough to measure variability; inspect slices and failures; decide against a rule written in advance; and let reviewed production failures grow the suite. The same machinery applies whether the system produces one label or a hundred-step trajectory.

An implementation should preserve that contract instead of dictating it. For standardized model tasks, EleutherAI’s Language Model Evaluation Harness provides versionable task definitions, test instructions saved with revision history; model adapters, translation layers that let one test runner communicate through different model interfaces, the agreed ways software sends requests and receives results; per-sample logs, one evidence record for each case; and a reusable task library^[10]. OpenAI Evals combines a framework, reusable code and conventions for building tests, with an open benchmark registry, a public catalog of contributed tests, and supports custom system evaluations^[11]. Inspect AI composes datasets with solvers, the procedures that attempt each task; tools; sandboxes, isolated computing environments where unfamiliar or potentially unsafe actions can run without affecting the host system; and scorers for model, workflow, and tool-using evaluations^[12]. Promptfoo offers declarative comparisons, tests described through named settings and conditions rather than code that directs every execution step; assertions, explicit conditions an output must satisfy; red-team tests, adversarial probes for safety or security weaknesses; and continuous integration, usually shortened to CI, which runs checks automatically when code changes^[13]. Reading these projects’ task schemas, the required fields used to describe a test, and result artifacts, the reports and records produced by a run, is valuable even when a small product eval only needs a test runner, software that executes cases and collects results, a dataset file, and a report.