Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation

AI summary

Overview

Research area: Natural Language Processing, specifically mechanistic interpretability of large language model evaluators (LLM-as-a-judge) for text summarization quality.

Technical level: Advanced. The paper assumes familiarity with transformer internals (residual streams, MLP sublayers, attention output projections), causal tracing / activation patching, logit lens, and attention-head knockout.

Scope: A mechanistic case study of how two open-source LLM judges — Themis (Llama-3-8B) and Prometheus (Mistral-7B, specifically Prometheus-7B-v2.0) — internally compute summary quality ratings, using a purpose-built eight-attack perturbation taxonomy and a four-experiment interpretability battery.

What This Paper Is About

LLM-based evaluators are now widely used both as scoring tools and as automated training signals for natural language generation, yet prior work has almost entirely examined their behavior (agreement with human ratings, known biases, failure modes) rather than the internal computation that produces a score. This paper asks whether the model actually identifies the specific quality defect in a summary, or whether it emits a rating consistent with training-distribution patterns without engaging the underlying error. To answer this, the authors build paired clean and corrupted summaries with controlled error intensity and explicit token-level modification maps, then trace how the rating is computed inside Themis and Prometheus.

Key Contributions

1. An eight-attack perturbation taxonomy spanning the Readability and Adequacy dimensions of NLG quality (Readability: Preposition Mismatch, Tense Mismatch, Spelling, Sequential Reordering; Adequacy: Entity Swap, Numerical/Date Swap, Coreference Mismatch, Antonym & Negation), consolidated from over 35 error types drawn from prior perturbation benchmarks, plus an automated GPT-4o generation pipeline that returns explicit original_tokens and new_tokens arrays so no post-hoc alignment is needed.

2. The first mechanistic characterization of LLM-based NLG evaluation, applying four methods in combination — window-mode causal tracing, last-token-mode causal tracing, logit lens, and attention-head knockout — across MLP sublayers and attention output projections on two evaluators from different base-model families.

3. A two-stage pipeline account of rating computation, identifying where errors are registered, how the signal routes to the output position, when the decision crystallizes, and which heads implement the verdict.

4. A base-model control at the same scale (Llama-3-8B) that isolates the two specific mechanisms judge fine-tuning installs on top of an existing substrate, showing the evaluation pipeline is not built from scratch.

Main Findings

• Errors register locally in early MLPs. Window-mode MLP causal effects concentrate at relative position +0 in layers 0–5 and are near zero elsewhere on both Themis and Prometheus, with the strongest magnitudes for the most token-localized attacks (Entity, Numerical, Antonym). The perturbed token's MLP output encodes the corrupted content regardless of evaluation criterion.

• Attention strategy differs by criterion. Preposition effects spread from +0 through +2–3 across layers 0–10 (checking adjacent syntactic context); Entity and Numerical concentrate sharply at +0 in mid layers with no horizontal spread; Coreference, Antonym, Tense, and Reordering produce weak window-mode attention overall, with their comparison happening through last-position attention to distant tokens outside the six-token window. Readability-related errors spread across adjacent syntactic context, while adequacy attacks remain concentrated on the perturbed tokens.

• A shared routing-and-integration architecture at the final position. On Themis, last-position attention output-projection effects concentrate in a single band at L13–L15, after which the MLP effect rises in a stepped cascade with bands at L17–L18, L25–L27, and a peak at L31 acting as the final write into the rating-token direction. Prometheus shares the routing band but its last-position attention spans more depths (L14–L17, L19–L21, L29–L30), and its L31 MLP effect is strongly negative where Themis's is strongly positive.

• The rating decision crystallizes at a sharp late layer. Logit-lens probability mass on the five rating tokens is essentially zero through L≈22 on Themis and rises steeply over a narrow band, with every attack transitioning at the same depth: L=26 on Themis and L=25 on Prometheus. Bootstrapping the max-slope depth across seeds yields a 95% CI of [26, 26] on Themis and [25, 25] on Prometheus.

• An implicit severity hierarchy post-crystallization. On Themis, Entity and Antonym crystallize to Rating 2 (harshest), Tense to Rating 4 (mildest), and the remaining attacks cluster at Rating 3. Prometheus reproduces this ordering more punitively (Entity and Antonym at Rating 1, most others at Rating 4) but with less confidence: its corrupt-rating probability peaks at ~0.5 around L=30 and then declines, whereas Themis's climbs monotonically toward ~1.0 by the final layer. Prometheus also shows a sustained Rating-5 prior pre-crystallization that is never fully suppressed.

• A narrow band of attention heads at L10–L11 implements the verdict. Every individual attack panel on Themis shows a horizontal band at L10–L11 with alternating disruptive (red) and suppressive (blue) heads and negligible individual-head effects elsewhere. L10–L11 contains 30.2% ± 0.9% of Themis's ablation-active heads (a 4.8× enrichment over the uniform expectation from 32 layers), and L10–L15 contains 39.9% ± 0.5% of Prometheus's. Readability panels additionally recruit heads at L7–L9, while Adequacy panels remain sparse between L5 and L9. Both models show essentially no individual-head effects above L15.

• The pipeline generalizes across data domains. Replicating the full sweep on XSum preserves the crystallization depth exactly: Themis at L=26 (95% bootstrap CI [26, 26], N=2,242 pooled samples) and Prometheus at L=25 ([25, 25], N=2,829), with 89% to 100% of samples within ±1 layer of the median. Themis's two-stage MLP separation replicates with a mean ratio of 21.9× across 7 attacks with defined ratios (below-L15 = 0.007, above-L15 = 0.153). On Prometheus, below-L15 contribution becomes slightly negative across all attacks (−0.005 to −0.001) while above-L15 magnitude is preserved (≈0.09). Prometheus's attention peak shifts one layer up to L16, broadening the routing band to L13–L16. The L31 divergence is preserved: Themis writes positively (+0.26 to +0.50 across attacks), Prometheus negatively (−0.07 to −0.15).

• Fine-tuning sculpts an existing substrate rather than building a pipeline. The un-fine-tuned Llama-3-8B base model reproduces the routing architecture, the L13 attention band, and the L15 MLP peak, and its above-L15 MLP magnitude matches Themis's (0.131 vs. 0.125, mean across attacks × 3 seeds). But the two-stage separation is much weaker on the base model (below-L15 MLP effect 0.056 versus Themis's 0.010, a ~5.5× gap), and crystallization occurs two layers later at L=28 (95% bootstrap CI [28, 28], N=1,927 samples over 3 seeds). Base-model behavioral sensitivity is also lower: about 39% of pairs produce a rating change under single-instruction prompting, pooled across attacks.

• Per-attack Themis two-stage ratios (Table 2). Below-L15 / above-L15 / ratio: Entity 0.010 ± 0.001, 0.125 ± 0.010, 12.77×; Numerical 0.012 ± 0.002, 0.143 ± 0.014, 12.41×; Coreference 0.010 ± 0.001, 0.158 ± 0.008, 16.16×; Polarity 0.008 ± 0.001, 0.127 ± 0.010, 15.10×; Preposition 0.008 ± 0.003, 0.176 ± 0.006, 20.98×; Tense 0.013 ± 0.004, 0.177 ± 0.008, 14.19×; Spelling 0.010 ± 0.002, 0.165 ± 0.004, 15.76×; Reordering 0.007 ± 0.000, 0.137 ± 0.008, 20.13×. Mean: 0.010, 0.151, 15.56×, with crystallization at L26 [26, 26] for every attack.

Methodology in Plain English

The core obstacle is that causal tracing methods need paired "clean" and "corrupted" inputs with exactly tracked token-level edits, and existing adversarial summarization benchmarks introduce errors at the sentence or document level, too coarse to trace. So the authors designed their own perturbation framework with two controls: controlled intensity, which fixes the number of perturbed tokens per sample via a parameter k, and spatial localization, which records the positional indices of every perturbed token so causal tracing can anchor to known perturbation sites. Three criteria narrowed the design space to eight attacks: grounding in prior published benchmarks, compatibility with token-localizable tracing, and balanced coverage of four attacks per quality axis. Clean summaries come from CNN/DailyMail, with the same pipeline applied to XSum for a cross-domain test, and GPT-4o is prompted with category-specific few-shot exemplars to inject exactly k instances of a targeted error, returning a JSON-schema-constrained token map. Data quality was verified by inspecting roughly 20 clean/corrupt pairs per attack against three criteria, with typically 19 to 20 of 20 passing.

The analysis then proceeds in four experiments. Activation patching substitutes clean activations into a corrupt forward pass at specific (layer, position) sites and measures how much clean rating behavior is recovered, normalized so 1.0 recovers clean behavior and 0.0 matches the unmodified corrupt baseline; window-mode restores a six-token window starting at the perturbed token (relative positions +0 through +5) to localize where the perturbation is processed, while last-token mode restores only the final input position to localize at what depth the rating is assembled. Logit lens projects the residual stream at the final position through the final layer norm and unembedding at each layer to get a depth-resolved trajectory over the five rating tokens. Attention head knockout zeroes out each of the 32 × 32 = 1024 heads in a model during a corrupt forward pass, with positive effects identifying disruptive heads that drive the corrupt rating and negative effects identifying suppressive heads that defend the clean rating. Experiments use 200 clean/corrupt pairs per attack across all eight attacks, averaged over three seeds and using a 20% trimmed mean, with a behavioral filter that retains only pairs whose rating actually changes after perturbation. Both evaluators have 32 transformer layers and 32 attention heads per layer.

Why This Matters

Research impact: The paper opens a new object of mechanistic study — the evaluator itself. Prior interpretability work has targeted factual recall, indirect-object identification, and other narrowly scoped behaviors; this work extends the toolkit to NLG evaluation in combination across two base-model families. The finding that fine-tuning installs two specific, localizable mechanisms (below-L15 MLP suppression and a two-layer advance of crystallization depth) rather than assembling a pipeline suggests that targeted interventions at those two loci might reproduce judge behavior without full task-specific fine-tuning.

Real-world applications:

• Contrastive evaluator training and localized-evaluator distillation — the authors explicitly name these as downstream uses of the released taxonomy, generation pipeline, and behaviorally verified corpus.
• Reward modeling and preference labeling — LLM judges increasingly serve as automated signals for these, and understanding where the rating is computed bears directly on reliability.
• Diagnosing evaluator failure modes — knowing that Readability errors recruit wider early-layer attention while Adequacy errors stay concentrated suggests criterion-specific attention supervision could improve robustness.
• Benchmark and adversarial dataset design — the controlled-intensity, position-tracked perturbation pipeline offers a template for datasets built for interpretability rather than only behavioral benchmarking.

Industry relevance: Organizations deploying LLM-as-a-judge systems for model evaluation, red-teaming, or RLHF pipelines gain a mechanistic account of when a judge is genuinely engaging with an error versus pattern-matching, plus an argument that evaluator behavior can be modified surgically rather than through full fine-tuning.

Future Directions

• A matched base-model control on Mistral-7B. The authors note their two-mechanism decomposition is shown on the Llama-3-8B substrate; whether Mistral-7B fine-tuning installs analogous or different mechanisms requires a corresponding base-model control, which they leave to future work.
• Extending beyond summarization and English. The taxonomy is designed to be task-agnostic, but the identified circuit has not been directly verified on dialogue response generation, story generation, or factuality assessment, nor on larger or multilingual evaluator models.
• Higher intensities and mixed attacks. All experiments use single-attack samples at intensity k = 1; behavior under higher intensities or two-or-more co-occurring perturbation categories is not characterized.
• Analyzing evaluator misses. The behavioral filter restricts analysis to samples where the rating demonstrably changes, so cases where the evaluator fails to detect a perturbation — which the authors say carry interpretable signal about robustness — fall outside the present analysis. Prompt-variation stability of the circuit is also flagged as an open empirical question.

Target Audience

Mechanistic interpretability researchers and NLP evaluation practitioners with a working knowledge of transformer internals; most valuable to those building or auditing LLM-as-a-judge systems, designing adversarial NLG benchmarks, or working on reward modeling and evaluator distillation. Readers seeking a behavioral comparison of evaluator accuracy against human judgments will find that framing deliberately outside this paper's scope, since its stated aim is the internal procedure rather than agreement measurement.

Authors’ abstract

LLM-based evaluators of natural language generation (NLG) quality are widely deployed as scoring tools and as automated training signals, yet the internal procedure by which they assign a rating remains poorly understood. We investigate this procedure mechanistically through an eight-attack perturbation taxonomy across the Readability and Adequacy dimensions of NLG quality, a generation pipeline that produces paired clean and corrupt summaries with controlled error intensity and explicit token-level modification maps, and a four-experiment battery of causal tracing, logit-lens vocabulary projection, and attention-head knockout applied to Themis (Llama-3-8B) and Prometheus (Mistral-7B). Both evaluators implement a structured, coherent evaluation pipeline operating in two stages: below layer 15, attention performs local error comparison and routes the result to the final input position; above it, the MLP cascade integrates the signal and writes the rating, with the decision crystallizing in the residual stream at a sharp late layer (L = 26 on Themis, L = 25 on Prometheus). Furthermore, a base-model control at the same scale (Llama-3-8B) reproduces the routing architecture and crystallization but not the stage separation, isolating the two mechanisms that fine-tuning specifically installs, suppression of below-L15 MLP contribution at the last position and a two-layer advance of the crystallization depth, indicating that fine-tuning sculpts an existing substrate rather than building the pipeline from scratch. We release the source code and data at https://github.com/himil-v/judge-mech