Generative AI
Evaluating Generative AI Systems
Build representative, multidimensional, and continuously maintained evaluations for generative AI products and their complete execution traces.
By the end you can
- Translate product requirements into measurable evaluation dimensions and release criteria
- Construct representative datasets with normal, edge, adversarial, and regression cases
- Combine deterministic checks, model graders, human judgment, and production outcomes
- Diagnose component failures using traces rather than relying on one aggregate score
A single score cannot describe a generative product
A support assistant can be fluent, grounded, safe, fast, and affordable in different proportions. The average score can improve while a rare billing workflow becomes dangerously worse.
Evaluation begins by naming the decisions, populations, harms, and acceptable tradeoffs. Metrics are evidence for those questions, not substitutes for them.
The rest of this lesson leans on evaluations whose authors published their apparatus: how many scenarios, how many models, how many labellers, how much of the grid was actually measured. That apparatus is what a release decision rests on. The headline number is not.
The evaluation portfolio should mirror the product contract and its failure costs.
Predict first
How much of the map had anyone measured?
By 2022, language models were evaluated paper by paper, each paper choosing its own benchmarks. A Stanford team defined 16 core scenarios (question answering, summarisation, toxicity detection and others) and then compiled every evaluation it could find for the prominent models of the day, to see how much of that grid had already been filled in.
Make a prediction before reading on. On average, on what share of the 16 core scenarios had those models previously been evaluated: most of them, about half, or less than a fifth?
Settle on an answer, then read on — the next section is the reveal
Case
Thirty models, forty-two scenarios, seven metrics each
HELM is the worked example, and its own numbers are the argument. Stanford's Center for Research on Foundation Models published it in November 2022. The design refuses the single score from the outset. Seven metrics — accuracy, calibration, robustness, fairness, bias, toxicity and efficiency — measured on each of 16 core scenarios, a combination the setup made possible 87.5% of the time. Seven metrics across sixteen scenarios is up to 112 measurements for one model. Accuracy is one column of that grid, not the grid itself. Thirty prominent language models were then run under standardised conditions on all 42 scenarios: the 16 core ones plus 26 targeted ones.
What the grid exposed was not the ranking. It was the state of the evidence behind every ranking that came before it. “Prior to HELM, models on average were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common,” the abstract reports. HELM raised that coverage to 96.0%.
Read the two figures together: 17.9% before, 96.0% after. Where coverage is that thin, a comparison between two systems is partly an artefact of what each team chose to measure and report. Standardising the scenarios, and reporting every metric on every one of them, is what turns a table of numbers into a comparison at all.
FigureInteractive
A full-width proportion strip carries the HELM finding that before HELM, models had on average been evaluated on just 17.9% of the core HELM scenarios — leaving 82.1% never evaluated. Below, seven labelled blocks name the metrics HELM measures on every scenario: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency, so 7 × 16 = 112 measurements for a single model, and accuracy is one column of it. Three figures close it: 30 models run under standardised conditions on all 42 scenarios, and LLM judges matching human preferences over 80% of the time.
Visual
A release decision combines several forms of evidence
Each layer answers a different question about the system, and the top layer exists because the lower ones cannot stand in for it.
Twelve risks are unique to generative AI or made worse by it: CBRN information or capabilities, confabulation, dangerous/violent/hateful content, data privacy, environmental impacts, harmful bias or homogenization, human-AI configuration, information integrity, information security, intellectual property, obscene/degrading/abusive content, and value chain and component integration. That list is NIST's, from its 2024 generative AI profile, AI 600-1. Twelve categories is already more than one dimension of a benchmark score can carry.
The profile is blunt about the limits of the lower layers. Its appendix "Pre-Deployment Testing" has a subsection on the "Limitations of Current Pre-deployment Test Approaches", and it reads: “Current testing approaches often remain focused on laboratory conditions or restricted to benchmark test datasets and in silico techniques that may not extrapolate well to—or directly assess GAI impacts in real-world conditions.” That is why the stack does not stop at the offline suite. The production layer is not a nice-to-have on top of benchmark evidence. It is the only layer that observes the conditions the other four are proxies for.
- 01
Deterministic validity
Schema, syntax, tool arguments, calculations, citations, and policy invariants.
- 02
Task quality
Correctness, completeness, relevance, groundedness, and appropriate abstention.
- 03
Behavioral robustness
Paraphrases, contrast cases, long context, adversarial input, and distribution slices.
- 04
Human judgment
Rubric-based review of meaning, usefulness, tone, and ambiguous cases.
- 05
Production outcomes
Resolution, escalation, rework, complaints, harm, latency, and cost.
Comparison
Evaluation methods have different authority
Use complementary methods, and validate automated graders against trusted judgments. Two published measurements show why neither half of that instruction is rhetorical.
Order sensitivity is not a caveat. It is a number. The MT-Bench paper measured it directly, and its Table 2, "Position bias of different LLM judges", defines the quantity plainly: “Consistency is the percentage of cases where a judge gives consistent results when swapping the order of two assistants.” Under the paper's default prompt, GPT-4 gave consistent verdicts after swapping the two answers only 65.0% of the time, and favoured the first answer in 30.0% of cases. Claude-v1 was consistent just 23.8% of the time and favoured the first answer in 75.0% of cases. A separate "repetitive list" attack made Claude-v1 and GPT-3.5 judges fail on 91.3% of 23 answers, against 8.7% for GPT-4. A grader that changes its mind three times in four when you reverse the presentation order is measuring position as much as quality. Swapping or randomising order is a requirement, not a refinement.
Reviewer expertise is a design decision with a price tag attached, and GPQA priced it. David Rein and colleagues built a 448-question multiple-choice dataset in biology, physics and chemistry, and reported in 2023: “We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are "Google-proof").” The strongest GPT-4 baseline reached 39%. The same 448 items score 65% or 34% depending on who is answering them. "Human evaluation" without a stated reviewer population is not a specification of anything.
Deterministic checks
Apply rules when correctness can be computed exactly.
- Fast and reproducible
- Narrow semantic coverage
- Excellent for schemas and calculations
- Can miss plausible but wrong meaning
Model-based graders
Score open-ended outputs using a rubric and reference context.
- Scales semantic review
- Can share biases with the evaluated model
- Sensitive to prompt and order
- Must be calibrated on human-labeled cases
Human evaluation
Use trained reviewers for ambiguity, impact, and nuanced quality.
- Highest contextual flexibility
- Expensive and variable
- Needs clear rubrics and adjudication
- Should sample consequential and disagreement cases
Key idea
A model grader is a measurement instrument
Model graders can provide useful scale. But their verdicts depend on the rubric, prompt, reference evidence, ordering, and grader model, and they may prefer verbosity, familiar phrasing, or outputs from related model families. Measure agreement with trusted reviewers. Inspect disagreements, randomize order where relevant, and keep deterministic checks independent.
What that measurement looks like has been published. MT-Bench is 80 high-quality multi-turn questions across 8 categories, ten questions each. Its authors had 58 expert-level human labellers produce roughly 3K controlled votes, alongside 30K crowdsourced Chatbot Arena conversations. Against that reference they report: “Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.”
Notice what that sentence licenses and what it does not. Over 80% agreement, at the level humans reach with each other, on those 80 questions, with that judge, against that labelled reference. It is a calibration result, not a promise of ground truth, and it cost 58 expert labellers and 3K controlled votes to obtain. A grader score on your own product carries the authority of the equivalent exercise you have run, and no more. Where you have run none, the number is cheap in both senses.
Validate the evaluator before using it to validate the product.
Steps
Build an eval-driven development loop
Treat the suite as versioned evidence connected to every release. Two of these steps have been costed in public, and neither cost is optional.
Steps 2 and 4 — fresh cases, honest baselines. GSM1k is a 1,205-problem benchmark commissioned to match GSM8k on human solve rate, solution steps and answer magnitude. Its authors then moved the leading models across to it: “When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting across almost all model sizes.” The gap tracked exposure. There was a positive relationship, Spearman's r^2 = 0.36, between a model's probability of generating a GSM8k example and its GSM8k-to-GSM1k performance gap. Accuracy drops of up to 8% on a public benchmark can be a property of the benchmark's circulation rather than of the system. Only a matched, freshly written set makes the difference visible.
Step 6 — gating. In one jurisdiction it is a legal duty for one class of developer, providers of general-purpose AI models with systemic risk; for anyone else, such as a team gating its own assistant, the clause below is a model of practice rather than an obligation. The EU Artificial Intelligence Act, Regulation (EU) 2024/1689, was adopted in June 2024 and published in the Official Journal of the European Union on 12 July 2024. Article 55(1)(a) obliges providers of general-purpose AI models with systemic risk to “perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks;”. Two words in that clause do work a leaderboard never does: standardised, and documenting. The suite has to be reproducible by someone else, and the adversarial testing has to leave a record.
1. Define the contract
Name task, user, harm, rubric dimensions, and acceptance criteria.
2. Assemble representative cases
Cover traffic, critical slices, adversarial inputs, and incidents.
3. Select measurements
Combine exact checks, graders, humans, and downstream outcomes.
4. Establish baselines
Compare current production, simple workflows, and candidate variants.
5. Inspect traces
Attribute failures to retrieval, prompt, tool, model, state, or policy.
6. Gate and monitor
Block regressions, canary releases, and feed production cases back into the suite.
Example
The eval-driven loop, run on a billing-dispute assistant
One release decision for a customer-support assistant runs through the six steps here. GSM1k's fresh-set lesson and the gating duty come from the lesson above. The assistant, its cases and every count below are invented, and the rates are computed.
- Define the contract: the assistant decides whether a disputed charge is refundable, cites the invoice line it relied on, and hands any dispute over €500 to a person. The release needs at least 95% correct eligibility on billing disputes and zero unescalated disputes over €500.
- Assemble representative cases: 800 cases: 500 sampled from traffic, 120 billing disputes, 80 adversarial requests and 100 past incidents. Another 60 billing disputes are written fresh by the billing team and never shown to anyone tuning prompts, the GSM1k idea at product scale.
- Select measurements: amounts and dates are checked by code; tone is scored by a model grader that agrees with staff on a calibration sample; eligibility on the billing slice is scored by billing staff.
- Establish baselines: on the 120 billing cases the current production assistant gets 101 right (84.2%), a rules-and-templates workflow 98 (81.7%), and the candidate 112 (93.3%). On the 60 fresh cases the candidate gets 51 (85%). Eight points lost on unseen cases says some of the gain came from familiarity with the suite.
- Inspect traces: of the candidate's 8 billing failures, 5 retrieved the wrong invoice, 2 hit a tool timeout and 1 was the model's own arithmetic. Most of the fix belongs to retrieval, not to the prompt.
- Gate and monitor: at 93.3% against a 95% gate the candidate does not ship and no canary starts. The 8 failures and the 9 fresh-case failures join the suite as regression cases, so the next candidate is measured against them.
Analogy
A flight test program uses more than one instrument
Before an aircraft carries passengers, engineers inspect components, simulate edge conditions, fly test flights, and write down operating limits. None of it reduces to a single dashboard number. Generative systems need the same layered evidence and the same ability to localize a failure.
An airframe either holds or it does not. A language task usually has several acceptable answers, which puts rubrics and human judgment nearer the center of the test program. And, as GPQA showed, whose expertise sits in the loop is part of the instrument rather than a staffing detail.
Release evidence should combine exact tests, realistic scenarios, expert review, and monitored operation.
Evaluation is a living description of acceptable behavior
Models, prompts, sources, tools, users, and policies all change. The evaluation suite has to be versioned, reviewed, and refreshed, not treated as a one-time benchmark. The cost of not doing that has been measured on one of the field's most-cited benchmarks.
MMLU-Redux, presented at NAACL in 2025, re-annotated 5,700 MMLU questions across all 57 subjects. The estimate is that 6.49% of MMLU questions contain errors overall, and the errors are not spread evenly: “For example, we find that 57% of the analysed questions in the Virology subset contain errors.” Re-annotating also produced significant discrepancies against originally reported model performance. Published scores had, in part, been rewarding agreement with mislabelled answers. A suite nobody re-reads slowly becomes a specification of the wrong behavior. It goes on being cited while it does.
The next lesson focuses on unsupported claims and uncertainty. It turns evaluation findings into runtime choices about verification, abstention, and escalation.
Key takeaways
- Generative evaluation should mirror the product contract, user populations, harms, and operating constraints: HELM measured seven metrics on each of 16 core scenarios rather than reporting accuracy alone.
- Coverage is the precondition for comparison — before HELM, models had on average been evaluated on 17.9% of the core scenarios, a figure HELM raised to 96.0%.
- Deterministic checks, model graders, human review, and production outcomes provide different forms of evidence, and NIST AI 600-1 warns that benchmark and laboratory testing may not extrapolate to real-world conditions.
- Automated graders require calibration against trusted judgments: GPT-4 judges reached over 80% agreement with human preferences on MT-Bench, yet held their verdict after an answer swap only 65.0% of the time.
- Held-out, freshly written cases separate skill from exposure — GSM1k revealed accuracy drops of up to 8% that GSM8k concealed — and trace-level evaluation distinguishes model failures from retrieval, tool, policy, state, or interface failures.
- An eval suite is a versioned product asset that must be re-read as well as re-run: MMLU-Redux estimated 6.49% of MMLU questions contain errors, and 57% in the Virology subset.