SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges

AI summary

Overview

Research area: Natural Language Processing / LLM evaluation — specifically reliability testing of LLM judges (LLM-as-a-judge).

Technical level: Intermediate. Readers should be comfortable with the ideas of LLM-as-a-judge, behavioral testing, and metamorphic testing (relations that should hold between scores before and after a controlled input change).

Scope: The paper presents SANE, an agent-based pipeline that automatically derives scorer-specific metamorphic test recipes for LLM judges, validates them, and compares the resulting test suites against the Judge Reliability Harness (JRH) across five scoring domains.

What This Paper Is About

LLM judges are now used to score benchmarks, select models, curate training data, and gate deployments, but their scores are only trustworthy if the judge responds predictably to input changes that should or should not affect its criterion. Existing testing harnesses rely on fixed, human-authored perturbation libraries, so they can only test perturbations and scorer types their authors anticipated. The paper's goal is to automate the scorer-specific step — figuring out which capabilities matter for a given scorer and inventing the perturbations and expected score relations that test them — and then to validate those tests before they are used.

Key Contributions

1. Automated scorer-specific recipe ideation. SANE fixes six behavioral requirements (monotonicity, subadditivity, repeatability, robustness, dynamic range, calibration) as reusable agent instructions, then has an agent read each scorer's specification, implementation, and representative inputs to identify relevant capabilities and derive recipes and expected relations for them. Because recipes are generated per scorer, the aligned recipe inventory grows as scorers are added.

2. A formal, scorer-independent validation framework. The paper separates three validity questions — whether a recipe and its expected relation agree with scorer intent (A_r), whether a generated case actually realizes the requested perturbation (Z_t,r), and whether the expected relation applies to that realized case (I_t,r) — and defines joint validity as their product, with a strict-majority family rule.

3. Empirical demonstration across five domains. SANE is evaluated on faithfulness, fluency, coherence, professionalism, and toxicity scorers and detects scorer deficiencies in all five, including two domains where JRH detects none.

4. Judge-level human alignment meta-evaluation. Rather than comparing individual outputs to human labels, the paper asks whether a harness's pass rates rank 11 judge models consistently with their independently measured FaithBench performance.

Main Findings

• Aligned recipes scale with the scorer set: SANE produces 108 unique aligned recipes across the five scorers versus seven for JRH's fixed library. Enumerating scorer subsets, SANE's aligned recipe count rises from a mean of 21 for one scorer to 108 for all five; JRH rises from a mean of five to seven.

• More candidate pairs survive alignment: Of 133 SANE candidate scorer-recipe pairs, 108 align with scorer intent; JRH yields 35 candidate pairs from its seven unique recipe specifications, of which 25 align.

• More validated test families: SANE yields 77 validated test families from 133 candidates; JRH yields 21 from 35. The end-to-end validity rates are comparable (57.9% versus 60.0%), so the advantage comes from the larger aligned inventory rather than a higher per-candidate pass rate — the paper reports 3.7 times as many validated families.

• Fault detection is broader: 31 SANE families and 10 JRH families detect at least one scorer fault. SANE produces fault-detecting families for all five scorers; JRH produces none for professionalism or toxicity. For coherence, JRH detects faults in three of four validated families and SANE in four of eleven.

• Validity decomposition favors different stages: SANE has higher recipe alignment (Pr(A) = 0.812 versus 0.714), JRH has higher conditional case realization (Pr(Z|A) = 0.860 versus 0.708), relation applicability after realization is nearly equal (0.987 versus 0.986), and candidate-to-joint-valid yield is 0.568 for SANE versus 0.606 for JRH. The analysis covers 133 candidate recipes and 1,330 potential cases for SANE, and 35 and 350 for JRH.

• Strong human alignment in judge ranking: On faithfulness, SANE's test-suite pass rates rank 11 judge models consistently with FaithBench AUROC (Spearman ρ = 0.852, exact two-sided permutation p = 0.0015; Kendall τ_b = 0.734), while JRH does not (ρ = 0.100, p = 0.771; τ_b = 0.081). SANE separates the same models across a pass-rate span 2.87 times wider. Opus 5, which generated both test corpora, is excluded from the panel.

• Conclusions are threshold-robust: Sweeping the family threshold from τ = 5 through τ = 10 with n = 10 sampled cases, SANE retains more validated and fault-detecting families at every setting. At the strictest threshold (τ = 10) the counts narrow to 22 versus 14 validated families and 8 versus 7 fault-detecting families.

• Scorers were externally validated first: Faithfulness on FaithBench reaches AUROC 0.707 and AUPRC 0.473; toxicity on Civil Comments reaches AUROC 0.803 and AUPRC 0.264; fluency on SummEval reaches Spearman 0.331 and Kendall 0.313; coherence on SummEval reaches Spearman 0.553 and Kendall 0.472; professionalism on Stanford Politeness reaches Spearman 0.336, Kendall 0.259, and weighted Spearman 0.277.

Methodology in Plain English

The researchers gave an agent model (Claude Opus 5) a fixed checklist of six general properties any reliable scorer should have. For each scorer, the agent reads the scorer's written specification, its code and output contract, and some example inputs, then decides which concrete capabilities that scorer needs and invents paired tests for them: a text perturbation plus a rule for how the score should move. For example, for a toxicity scorer it identified "implicit toxicity without profanity" as a capability and generated a case that appends profanity-free belittling language, with the rule that the augmented text's toxicity score must not drop below the base score minus a noise tolerance.

Each recipe is then instantiated into concrete test cases. Before any scoring happens, three separate automated judges (three independent GPT-5.2 predictions, majority-voted) check whether the recipe matches scorer intent, whether the generated text really performs the requested change, and whether the expected relation makes sense for that specific case. Only cases passing all three checks count. A recipe becomes a validated test family if at least six of ten sampled cases pass (a strict majority).

The comparison baseline, JRH, was regenerated with the same Claude Opus 5 model for both generation and validation, with its released recipes, prompts, and expected relations preserved and all optional human review disabled. Both harnesses received the same balanced 1,200-example dataset: 300 examples each from AggreFact-CNN, AggreFact-XSum, TofuEval-MediaS, and TofuEval-MeetB. Faults are cases where the scorer output violates the expected relation. Finally, the faithfulness test suites' pass rates were compared against human-labeled FaithBench AUROC at the level of judge-model rankings.

Why This Matters

Impact on research: The paper reframes judge reliability testing as a generation-and-validation problem rather than a curated-library problem. It also introduces an explicit accounting of where automated test generation fails, which makes reliability claims auditable at the recipe, case, and instance level.

Real-world applications:

• Benchmark evaluation and model selection, where an unreliable judge silently reshuffles leaderboards.
• Data curation and supervised fine-tuning filtering, where judge scores decide which examples enter training.
• Preference and reward signal generation for post-training.
• Deployment gates for models and prompts, where judge decisions gate releases.

Industry relevance: Teams deploying LLM judges usually cannot hand-author perturbation suites for every custom rubric or scorer. SANE's premise — that the general requirements stay fixed while the recipes adapt to each scorer — offers a path to reliability coverage that scales with the number of scorers in production rather than with human authoring effort. The specific result that JRH's fixed library found no faults for professionalism or toxicity, while SANE did, is the practical case for scorer-specific generation.

Future Directions

• Weaker generator models. Both harnesses used the same generator, which controls for generator capability but leaves open whether absolute generation and realization rates transfer to weaker agent models.
• Human validation of the oracle. Human annotation of oracle decisions would give an independent estimate of oracle error, which may affect the reported scorer-deficiency counts.
• Alignment beyond faithfulness. Judge-ranking alignment was measured only for faithfulness, the criterion supported by both harnesses and by an independent human-labeled benchmark. Extending this requires comparable human benchmarks for other criteria.
• A retrieval library of validated designs. The authors propose reusing validated scorer problems, capabilities, and recipes across related scorer domains while preserving scorer-specific reasoning.

Target Audience

Evaluation engineers and ML reliability researchers who build or audit LLM-judge pipelines; benchmark maintainers who need to justify that a judge measures its stated criterion; trust-and-safety and model-release teams using judges as deployment gates; and practitioners who already use behavioral or metamorphic testing and want to see it applied to stochastic evaluator models.

Authors’ abstract

LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training. Their decisions are useful only when each judge responds reliably to changes affecting its scoring criterion. Existing methods for reliability testing perturb inputs with known expected effects on judge scores. Prominent LLM-judge testing harnesses combine human-authored perturbation recipes with synthetic test generation, leaving scorer-specific recipe ideation outside the automated pipeline. This limits their use to fixed perturbations and pre-assumed scorer types. We introduce SANE, an agent-based framework that encodes behavioral requirements for stochastic evaluators as reusable agent instructions. Given a scorer specification, implementation, and representative inputs, the pipeline first identifies scorer-specific capabilities, then automatically derives perturbation recipes and expected score relations for each. SANE also defines a scorer-independent validation framework for generated judge tests. It determines whether each recipe agrees with scorer intent and whether each instantiated case realizes the recipe and supports its expected relation. Because SANE generates a recipe inventory tailored to each scorer, the number of aligned recipes grows as scorers are added. Across five domains, SANE produces 3.7 times as many validated test families as the Judge Reliability Harness (JRH) at comparable end-to-end validity and detects scorer deficiencies in every domain, including professionalism and toxicity, where JRH detects none. On faithfulness, SANE achieves strong human alignment in judge ranking: its test-suite scores rank 11 judge models consistently with human-labeled FaithBench performance (Spearman rho = 0.852), while JRH does not (rho = 0.100).