Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry

AI summary

Overview

Research area: Natural language processing, specifically automated evaluation of language model outputs ("LLM-as-a-Judge") and interpretability methods such as linear probing of hidden representations.

Technical level: Intermediate. The high-level argument is intuitive, but the paper assumes familiarity with transformer hidden states, probing classifiers, and classification metrics like F1.

Scope: The paper argues that evaluation requires far less semantic capacity than generation, and demonstrates that lightweight probes on small language models' internal representations can approximate large-model judges for scoring reasoning outputs.

What This Paper Is About

The dominant way to evaluate model outputs without reference answers is to prompt a large proprietary model to act as a judge. This is expensive, depends heavily on prompt wording, and is opaque because the judge's internals are inaccessible. The authors ask whether small, open models can serve as evaluators not by generating verdicts, but by exposing the evaluative information already encoded in their hidden states — even when their text generation is poor.

Key Contributions

1. An empirical observation about small models: Despite weak generative performance on evaluation prompts, small LMs (0.6B–8B) contain strong evaluation-relevant signals in their intermediate hidden states.

2. The Semantic Capacity Asymmetry Hypothesis: A formalization of the claim that accurate evaluation demands significantly less semantic capacity than generation, and can be grounded in compressed intermediate representations rather than decoded text.

3. The Representation-as-a-Judge paradigm: A reframing of evaluation as a decoding-free process that probes internal model structure instead of prompting for output.

4. INSPECTOR: A concrete probing framework that extracts pooled layer representations, attention entropy statistics, and PCA-projected features from a frozen small LM, then trains lightweight classifiers to predict aspect-level evaluation scores. Demonstrated on GSM8K, MATH, and GPQA, and applied to filtering noisy data for supervised fine-tuning.

Main Findings

• Probing beats prompting by a wide margin. Across reasoning benchmarks, internal-representation probes improve weighted average F1 over prompt-based evaluation on the same small models by more than 20% on most tasks. Weak generated text does not mean the underlying knowledge is absent — it may simply be lost during decoding.

• Binary classification is reliable enough for data filtering. Multiclass score prediction remains modest (roughly 50–60% F1), but binary high/low-quality classification reaches 80–90%, making it viable as a coarse, reference-free filter.

• Bigger models are not automatically better judges. Qwen3-1.7B outperforms Llama-3.1-8B-Instruct on some benchmarks, and Llama-3.2-1B-Instruct beats Llama-3.1-8B-Instruct on binary fluency probing (96.32% vs. 92.65%). Model families have distinct aspect-level strengths, so scaling does not guarantee evaluation quality.

• Mean pooling plus logistic regression works best. The ablation shows averaged token vectors consistently outperform last-token, min, max, and concat pooling, and logistic regression outperforms Random Forests, MLPs, and SVMs — consistent with prior findings that simple probes match complex ones.

• Evaluative signal concentrates in mid-to-upper layers. Layer-wise analysis shows hidden states correlate substantially with judge scores, with the strongest signals in mid-to-upper layers rather than only at the output stage. PCA-derived feature subspaces reveal these signals better than scalar or attention-derived features.

• Probe-based filtering matches a strong LLM filter for SFT. Using Qwen3-1.7B probes to rank distillation data for Llama-2-7B-Chat yields training curves comparable to filtering with DeepSeek-V3, and both clearly beat random filtering. Performance follows an up–down–up pattern, indicating quality matters most at small data scales while quantity dominates as training sets grow.

Methodology in Plain English

The pipeline has three stages.

Annotation. A mid-sized model (Llama-3-8B-Instruct) generates responses to benchmark questions. This is deliberate: a less capable generator produces a diverse mix of good and bad responses, which is what the probes need to learn from. A strong model (DeepSeek-V3) then scores each (question, response) pair on a 1–5 scale across five aspects: semantic consistency, logicality, informativeness, fluency, and factuality. These scores become the gold labels. Because high scores dominate naturally, the authors downsample each score level to the smallest level's count to keep classes balanced.

Probing. Each evaluation prompt is fed to a frozen small LM. For every layer, the authors extract several pooled vectors (mean, last, min, max, concat), plus attention entropy summaries and statistics like norm, variance, and entropy. These are reduced with PCA and assembled into per-layer feature matrices. Simple logistic probes are trained with stratified cross-validation, and layers and pooling choices are ranked by how well they predict the gold scores. All data-dependent transforms (imputation, PCA, scaling) are applied inside the cross-validation loop to prevent leakage.

Classifier construction. The top-ranked layers are assembled iteratively — a layer is added only if it improves performance — and features are concatenated across layers. A grid search over classifiers and feature sets picks the final evaluator, with ties broken toward fewer layers and lower variance. The resulting classifier acts as a cheap surrogate for the large judge.

Why This Matters

For research, the paper challenges the assumption that good evaluation requires a large generative model. It introduces probing as a tool for evaluation rather than linguistic diagnostics, and it offers a mechanistic claim — evaluation needs less capacity than generation — that can be tested and refined. It also shows that internal representations carry information that surface outputs do not, connecting to existing work on latent knowledge. A secondary benefit is interpretability: evaluation decisions come from inspectable linear classifiers over named layers and features, unlike an opaque prompted judgment.

Real-world applications:

• Training data curation: Filtering large instruction or reasoning datasets before supervised fine-tuning, reducing dependence on paid API judges.
• Reinforcement learning from feedback: Serving as a cheap reward or filtering signal during iterative model training.
• Evaluation in regulated or private settings: Running quality checks on-premises with small open models, where sending data to a proprietary judge is not viable.
• Continuous evaluation pipelines: Scoring model outputs at scale without the cost and latency of autoregressive decoding from a large judge.

Industry relevance is direct: evaluation cost is a real bottleneck for teams iterating on models, and this approach reduces it to a forward pass plus a linear classifier over cached hidden states. The authors note the method's practical reproducibility, releasing code and data.

Future Directions

• Standardizing evaluation aspects. The authors acknowledge the five aspects are inherited from prior work and may not be optimal — fluency in particular is too easy, with most responses scoring high. Better aspect design is an open question.
• Broadening beyond math and science. The experiments cover GSM8K, MATH, and GPQA. Commonsense reasoning and code generation remain untested.
• Removing dependence on a single judge. All gold labels come from DeepSeek-V3, which introduces potential bias into the probes. Comparing judges from different organizations would test robustness.
• Closing the multiclass gap. Binary classification is reliable but multiclass scores are only 50–60% F1. Whether better probes, larger probing data, or layer-fusion strategies can recover finer-grained judgments is unresolved.

Target Audience

Researchers working on LLM evaluation, interpretability, and probing methods; engineers building data curation or RLHF pipelines who want cheaper alternatives to proprietary judges; and practitioners interested in what small language models actually encode internally versus what they can produce. It is less suited to readers looking for beginner-level tutorials, since the methodology assumes comfort with transformer internals and classifier evaluation.

Authors’ abstract

Large language models (LLMs) are widely used as reference-free evaluators via prompting, but this "LLM-as-a-Judge" paradigm is costly, opaque, and sensitive to prompt design. In this work, we investigate whether smaller models can serve as efficient evaluators by leveraging internal representations instead of surface generation. We uncover a consistent empirical pattern: small LMs, despite with weak generative ability, encode rich evaluative signals in their hidden states. This motivates us to propose the Semantic Capacity Asymmetry Hypothesis: evaluation requires significantly less semantic capacity than generation and can be grounded in intermediate representations, suggesting that evaluation does not necessarily need to rely on large-scale generative models but can instead leverage latent features from smaller ones. Our findings motivate a paradigm shift from LLM-as-a-Judge to Representation-as-a-Judge, a decoding-free evaluation strategy that probes internal model structure rather than relying on prompted output. We instantiate this paradigm through INSPECTOR, a probing-based framework that predicts aspect-level evaluation scores from small model representations. Experiments on reasoning benchmarks (GSM8K, MATH, GPQA) show that INSPECTOR substantially outperforms prompting-based small LMs and closely approximates full LLM judges, while offering a more efficient, reliable, and interpretable alternative for scalable evaluation. The code and data are available at: https://github.com/zhuochunli/Representation-as-a-judge