Generative AI
Evaluating Generative AI Systems
Build representative, multidimensional, and continuously maintained evaluations for generative AI products and their complete execution traces.
By the end you can
- Translate product requirements into measurable evaluation dimensions and release criteria
- Construct representative datasets with normal, edge, adversarial, and regression cases
- Combine deterministic checks, model graders, human judgment, and production outcomes
- Diagnose component failures using traces rather than relying on one aggregate score
A single score cannot describe a generative product
A support assistant can be fluent, grounded, safe, fast, and affordable in different proportions. The average score can improve while a rare billing workflow becomes dangerously worse.
Evaluation begins by naming the decisions, populations, harms, and acceptable tradeoffs. Metrics are evidence for those questions, not substitutes for them.
The rest of this lesson leans on evaluations whose authors published their apparatus: how many scenarios, how many models, how many labellers, how much of the grid was actually measured. That apparatus is what a release decision rests on. The headline number is not.
The evaluation portfolio should mirror the product contract and its failure costs.
Case
Thirty models, forty-two scenarios, seven metrics each
HELM is the worked example, and its own numbers are the argument. Stanford's Center for Research on Foundation Models published it in November 2022. The design refuses the single score from the outset. Seven metrics — accuracy, calibration, robustness, fairness, bias, toxicity and efficiency — measured on each of 16 core scenarios, a combination the setup made possible 87.5% of the time. Seven metrics across sixteen scenarios is 112 measurements for one model. Accuracy is one column of that grid, not the grid itself. Thirty prominent language models were then run under standardised conditions on all 42 scenarios: the 16 core ones plus 26 targeted ones.
What the grid exposed was not the ranking. It was the state of the evidence behind every ranking that came before it. “Prior to HELM, models on average were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common,” the abstract reports. HELM raised that coverage to 96.0%.
Read the two figures together: 17.9% before, 96.0% after. Where coverage is that thin, a comparison between two systems is partly an artefact of what each team chose to measure and report. Standardising the scenarios, and reporting every metric on every one of them, is what turns a table of numbers into a comparison at all.
Figure
Visual
A release decision combines several forms of evidence
Each layer answers a different question about the system, and the top layer exists because the lower ones cannot stand in for it.
Twelve risks are unique to generative AI or made worse by it: CBRN information or capabilities, confabulation, dangerous/violent/hateful content, data privacy, environmental impacts, harmful bias or homogenization, human-AI configuration, information integrity, information security, intellectual property, obscene/degrading/abusive content, and value chain and component integration. That list is NIST's, from its 2024 generative AI profile, AI 600-1. Twelve categories is already more than one dimension of a benchmark score can carry.
The profile is blunt about the limits of the lower layers. Its appendix "Pre-Deployment Testing" has a subsection on the "Limitations of Current Pre-deployment Test Approaches", and it reads: “Current testing approaches often remain focused on laboratory conditions or restricted to benchmark test datasets and in silico techniques that may not extrapolate well to—or directly assess GAI impacts in real-world conditions.” That is why the stack does not stop at the offline suite. The production layer is not a nice-to-have on top of benchmark evidence. It is the only layer that observes the conditions the other four are proxies for.
- 01
Deterministic validity
Schema, syntax, tool arguments, calculations, citations, and policy invariants.
- 02
Task quality
Correctness, completeness, relevance, groundedness, and appropriate abstention.
- 03
Behavioral robustness
Paraphrases, contrast cases, long context, adversarial input, and distribution slices.
- 04
Human judgment
Rubric-based review of meaning, usefulness, tone, and ambiguous cases.
- 05
Production outcomes
Resolution, escalation, rework, complaints, harm, latency, and cost.
Comparison
Evaluation methods have different authority
Use complementary methods, and validate automated graders against trusted judgments. Two published measurements show why neither half of that instruction is rhetorical.
Order sensitivity is not a caveat. It is a number. The MT-Bench paper measured it directly, and its Table 2, "Position bias of different LLM judges", defines the quantity plainly: “Consistency is the percentage of cases where a judge gives consistent results when swapping the order of two assistants.” Under the paper's default prompt, GPT-4 gave consistent verdicts after swapping the two answers only 65.0% of the time, and favoured the first answer in 30.0% of cases. Claude-v1 was consistent just 23.8% of the time and favoured the first answer in 75.0% of cases. A separate "repetitive list" attack made Claude-v1 and GPT-3.5 judges fail on 91.3% of 23 answers, against 8.7% for GPT-4. A grader that changes its mind three times in four when you reverse the presentation order is measuring position as much as quality. Swapping or randomising order is a requirement, not a refinement.
Reviewer expertise is a design decision with a price tag attached, and GPQA priced it. David Rein and colleagues built a 448-question multiple-choice dataset in biology, physics and chemistry, and reported in 2023: “We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are "Google-proof").” The strongest GPT-4 baseline reached 39%. The same 448 items score 65% or 34% depending on who is holding the rubric. "Human evaluation" without a stated reviewer population is not a specification of anything.
Deterministic checks
Apply rules when correctness can be computed exactly.
- Fast and reproducible
- Narrow semantic coverage
- Excellent for schemas and calculations
- Can miss plausible but wrong meaning
Model-based graders
Score open-ended outputs using a rubric and reference context.
- Scales semantic review
- Can share biases with the evaluated model
- Sensitive to prompt and order
- Must be calibrated on human-labeled cases
Human evaluation
Use trained reviewers for ambiguity, impact, and nuanced quality.
- Highest contextual flexibility
- Expensive and variable
- Needs clear rubrics and adjudication
- Should sample consequential and disagreement cases
Key idea
A model grader is a measurement instrument
Model graders can provide useful scale. But their verdicts depend on the rubric, prompt, reference evidence, ordering, and grader model, and they may prefer verbosity, familiar phrasing, or outputs from related model families. Measure agreement with trusted reviewers. Inspect disagreements, randomize order where relevant, and keep deterministic checks independent.
What that measurement looks like has been published. MT-Bench is 80 high-quality multi-turn questions across 8 categories, ten questions each. Its authors had 58 expert-level human labellers produce roughly 3K controlled votes, alongside 30K crowdsourced Chatbot Arena conversations. Against that reference they report: “Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.”
Notice what that sentence licenses and what it does not. Over 80% agreement, at the level humans reach with each other, on those 80 questions, with that judge, against that labelled reference. It is a calibration result, not a promise of ground truth, and it cost 58 expert labellers and 3K controlled votes to obtain. A grader score on your own product carries the authority of the equivalent exercise you have run, and no more. Where you have run none, the number is cheap in both senses.
Validate the evaluator before using it to validate the product.
Steps
Build an eval-driven development loop
Treat the suite as versioned evidence connected to every release. Two of these steps have been costed in public, and neither cost is optional.
Steps 2 and 4 — fresh cases, honest baselines. GSM1k is a 1,000-problem benchmark commissioned to match GSM8k on human solve rate, solution steps and answer magnitude. Its authors then moved the leading models across to it: “When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting across almost all model sizes.” The gap tracked exposure. There was a positive relationship, Spearman's r^2 = 0.36, between a model's probability of generating a GSM8k example and its GSM8k-to-GSM1k performance gap. Up to eight points of a public benchmark score can be a property of the benchmark's circulation rather than of the system. Only a matched, freshly written set makes the difference visible.
Step 6 — gating. In one jurisdiction it is a legal duty rather than team hygiene. The EU Artificial Intelligence Act, Regulation (EU) 2024/1689, was adopted in June 2024 and published in the Official Journal of the European Union on 12 July 2024. Article 55(1)(a) obliges providers of general-purpose AI models with systemic risk to “perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks;”. Two words in that clause do work a leaderboard never does: standardised, and documenting. The suite has to be reproducible by someone else, and the adversarial testing has to leave a record.
1. Define the contract
Name task, user, harm, rubric dimensions, and acceptance criteria.
2. Assemble representative cases
Cover traffic, critical slices, adversarial inputs, and incidents.
3. Select measurements
Combine exact checks, graders, humans, and downstream outcomes.
4. Establish baselines
Compare current production, simple workflows, and candidate variants.
5. Inspect traces
Attribute failures to retrieval, prompt, tool, model, state, or policy.
6. Gate and monitor
Block regressions, canary releases, and feed production cases back into the suite.
Analogy
A flight test program uses more than one instrument
Before an aircraft carries passengers, engineers inspect components, simulate edge conditions, fly test flights, and write down operating limits. None of it reduces to a single dashboard number. Generative systems need the same layered evidence and the same ability to localize a failure.
An airframe either holds or it does not. A language task usually has several acceptable answers, which puts rubrics and human judgment nearer the center of the test program. And, as GPQA showed, it makes the choice of who judges part of the instrument rather than a staffing detail.
Release evidence should combine exact tests, realistic scenarios, expert review, and monitored operation.
Evaluation is a living description of acceptable behavior
Models, prompts, sources, tools, users, and policies all change. The evaluation suite has to be versioned, reviewed, and refreshed, not treated as a one-time benchmark. The cost of not doing that has been measured on one of the field's most-cited benchmarks.
MMLU-Redux, presented at NAACL in 2025, re-annotated 5,700 MMLU questions across all 57 subjects. The estimate is that 6.49% of MMLU questions contain errors overall, and the errors are not spread evenly: “For example, we find that 57% of the analysed questions in the Virology subset contain errors.” Re-annotating also produced significant discrepancies against originally reported model performance. Published scores had, in part, been rewarding agreement with mislabelled answers. A suite nobody re-reads slowly becomes a specification of the wrong behavior. It goes on being cited while it does.
The next lesson focuses on unsupported claims and uncertainty. It turns evaluation findings into runtime choices about verification, abstention, and escalation.
Key takeaways
- Generative evaluation should mirror the product contract, user populations, harms, and operating constraints: HELM measured seven metrics on each of 16 core scenarios rather than reporting accuracy alone.
- Coverage is the precondition for comparison — before HELM, models had on average been evaluated on 17.9% of the core scenarios, a figure HELM raised to 96.0%.
- Deterministic checks, model graders, human review, and production outcomes provide different forms of evidence, and NIST AI 600-1 warns that benchmark and laboratory testing may not extrapolate to real-world conditions.
- Automated graders require calibration against trusted judgments: GPT-4 judges reached over 80% agreement with human preferences on MT-Bench, yet held their verdict after an answer swap only 65.0% of the time.
- Held-out, freshly written cases separate skill from exposure — GSM1k revealed accuracy drops of up to 8% that GSM8k concealed — and trace-level evaluation distinguishes model failures from retrieval, tool, policy, state, or interface failures.
- An eval suite is a versioned product asset that must be re-read as well as re-run: MMLU-Redux estimated 6.49% of MMLU questions contain errors, and 57% in the Virology subset.