Judging in Latent Space: Efficient Generative Reward Modeling via Semantics-Preserving Compression

AI summary

Overview

Research area: Natural Language Processing, specifically reward modeling and LLM-as-judge evaluation, drawing on continuous latent reasoning and knowledge distillation for large language models.

Technical level: Advanced. The paper assumes familiarity with autoregressive language modeling, reward models, majority-vote decoding, KL-divergence distillation objectives, and inference-serving systems such as vLLM.

Scope: The paper introduces LatentGRM, a framework that replaces explicit, token-by-token rubric evaluation traces in generative reward models with compact continuous latent trajectories, and evaluates whether judgment accuracy survives the compression while inference cost falls.

What This Paper Is About

Generative reward models judge which of two candidate responses is better by writing out a full evaluation, often organized around a rubric of explicit criteria such as correctness, instruction compliance, and clarity. Verbalizing that reasoning for every criterion makes evaluation trajectories long and decoding expensive, and reward models are invoked repeatedly during policy optimization and response selection. The paper asks whether the intermediate evaluation computation can be carried out in a compact continuous latent space instead, while preserving multi-criterion preference information, and whether the compressed evaluation can later be read back out as text for inspection.

Key Contributions

1. LatentGRM framework. The authors present what they state is the first framework to bring continuous latent reasoning to reward modeling, combining rubric-guided semantic chunking, autonomous latent evaluation, and a separate interpreter for retrospective reconstruction.

2. Matched-baseline accuracy and efficiency evidence. Under matched training data and backbones, they report competitive aggregate preference accuracy at both 4B and 8B scales alongside substantial reductions in trajectory length and total judge inference time.

3. Analysis of information retained under compression. Through controlled rubric interventions and criterion-aligned reconstruction, they show that latent trajectories carry criterion-dependent preference information and that the Latent Trace Interpreter can recover criterion-level supporting justifications from teacher-derived trajectories.

4. Downstream utility check. Recovered evaluations are tested as feedback for revising responses, where the interpreter-restored trajectory is compared against explicit Rubric-RM textual feedback.

Main Findings

• Competitive aggregate accuracy at vote@5. LatentGRM scores 67.8 (4B) and 69.0 (8B) on Avg.-8, the unweighted mean of eight benchmark domains, compared with 67.0 and 68.1 for the matched explicit Rubric-RM. Its largest advantage is on Focus, at +6.0 and +6.4 points; across the other seven domains, Chat Hard and IFBench favor LatentGRM at both scales, RewardBench Chat favors Rubric-RM, and Precise IF changes direction between 4B and 8B, while PPE, RM-Bench Chat, and HelpSteer3 differ by at most 0.4 points.

• Voting helps latent judges. Majority voting over five rollouts raises LatentGRM's Avg.-8 from 65.6 to 67.8 at 4B (+2.2) and from 66.7 to 69.0 at 8B (+2.3).

• Trajectory compression and speedups. At vote@5, LatentGRM-8B reduces generated positions by 8.87–9.23 times across RB Chat (716 examples), RB Chat Hard (912), RB2 Precise IF (960), and RB2 Focus (2970). This yields 8.17–8.82 times speedup under ordinary Transformers/PyTorch generation and 6.11–6.95 times under vLLM on the same eight-GPU configuration. The abstract frames the headline range as 8.9–9.2 times trajectory compression and 6.1–7.0 times total judge inference time reduction.

• Speedups persist across voting budgets. LatentGRM remains faster across vote@1–9, with 6.9–7.8 times speedups at vote@9. It scores higher on Chat Hard and Focus at every voting budget, remains below Rubric-RM on Chat, and varies by budget on Precise IF.

• Latent reasoning adds no per-step overhead. An equal-length control forcing both judges to generate exactly 50 positions yields only a 1.017 times runtime difference.

• Semantic Chunking beats fixed-rate chunking on instruction-sensitive tasks. On Qwen3-8B at vote@5 with the same compression rate rho = 8, Semantic Chunking gains 2.2, 3.9, and 1.7 points on Chat Hard, Precise IF, and Focus respectively, while Chat decreases by 0.7 points and the four-domain mean rises by 1.8 points.

• Criterion changes move preferences through the latent sequence. Across 398 RewardBench pairs (198 Chat, 200 Chat Hard), a criterion intervention shifts preference toward the designated response, reaching a mean log-odds change of 4.58 at the trajectory endpoint. When the evaluation prompt is held fixed to the baseline rubric and only the latent-trajectory source varies, the criterion-intervention source still reaches a mean log-odds change of 2.75 at the endpoint, while the wording-control source remains much smaller.

• Latent trajectories are largely reconstructable. On 607 records containing 9,992 criteria (9,768 with explicit states), the Latent Trace Interpreter reaches 98.93 State Macro-F1 and 93.98 Vector Exact, versus 69.42 and 33.03 for Replay-hidden and 40.20 and 18.61 for Prompt-only. Prompt-only reaches 63.36 ROUGE-L despite its 40.20 State Macro-F1, which the authors use to argue that lexical overlap misses criterion-state errors. LTI also leads on Rubric Semantic (74.65) and ROUGE-L (79.24).

• Restored feedback is usable downstream. On 240 RewardBench Rubric examples with a frozen Qwen3-8B Base reviser, Interpreter-restored trajectory feedback yields the best result in every reported column: 80.49% Repair, 1.86% Regression, 57.50% Full, and 90.12% criterion satisfaction, versus 78.71, 4.65, 56.67, and 87.78 for explicit Rubric-RM trajectory feedback and 54.07, 5.20, 34.17, and 76.28 for Prompt-only. Repair is computed over 897 originally unsatisfied criteria and Regression over 1,076 originally satisfied criteria.

• The latent interface works before adaptation. Even without training the reviser to consume it, the Latent-only soft top-10 interface improves Repair by 9.25 points and Full by 7.50 points over Prompt-only.

Methodology in Plain English

The setup is pairwise preference judging. Each input contains a user request, a task-specific rubric, and two candidate responses, plus a training-time explicit evaluation trace and a binary preference label. An ordinary explicit judge writes out the trace as text between thinking markers and then emits a verdict.

LatentGRM replaces that text with a small number of continuous states. It builds on Latent-SFT's vocabulary-space distillation: at each compression boundary, an encoder reads the input and the explicit trace prefix and produces a hidden state, which is projected through the decoder's frozen output vocabulary and normalized over the top-K tokens. Those weighted tokens are converted back into a single continuous embedding through the decoder's frozen input embedding table, so the state can be inserted directly into the decoder's input sequence.

Stage 1 divides each evaluation trace into N spans using Semantic Chunking. Rubric-guided evaluations naturally split into compliance checking, criterion-specific analysis, and final synthesis. A budget allocator assigns each rubric block a fixed number of latent positions so the total equals the ceiling of M divided by the compression rate rho, and then linguistic evidence, including sentence, clause, and dependency cues, refines where the boundaries fall inside each block. Protected expressions such as response headers, criterion identifiers, and status statements are kept intact. The decoder is trained to recover the remaining evaluation text and the final preference from the latent prefix.

Stage 2 turns the exported distributions into an autonomous judge. Training adds Gumbel noise to the target distributions, renormalizes within the top-K support, and conditions each prediction on preceding perturbed target embeddings. The objective combines a KL term matching each perturbed latent target with a cross-entropy term predicting the end marker, the preference label, and EOS from the full trajectory. At inference the judge feeds back its own predictions, retaining and renormalizing top-K probabilities and applying the same Gumbel perturbation to form each next state; generation stops when the unperturbed argmax is the end-of-thinking marker, after which the preference is decoded greedily. The authors extended vLLM to accept this soft-embedding feedback while keeping continuous batching, KV caching, and tensor parallelism.

Separately, the Latent Trace Interpreter reads a completed trajectory using the same vocabulary-mixture construction and reconstructs the full evaluation text from its beginning, excluding the preference label. It runs offline and is never part of the online judgment path.

Evaluation uses 35,612 OpenRubrics training records, with Qwen3-4B or Qwen3-8B backbones and rubrics generated once offline by OpenRubrics generator-8B. Eight benchmark domains are used: RewardBench Chat and Chat Hard, PPE-IFEval, IFBench, RM-Bench Chat, RewardBench 2 Precise IF and Focus, and HelpSteer3. Every model is evaluated in both response orders and the resulting scores are averaged.

Why This Matters

Impact on research. The paper connects two previously separate lines of work, rubric-based reward modeling and vocabulary-space latent reasoning, and tests that combination under matched data and matched backbones rather than against weaker baselines. It also supplies an analytical handle on compression: because rubrics define criterion-level units, the authors can intervene on a single criterion and measure where in the trajectory the resulting preference shift appears, and can then check how much of that information survives a readout.

Real-world applications.

• Policy optimization, where reward models are queried repeatedly and inference cost compounds across training steps.
• Response selection or reranking at serving time, where judges must score many candidate pairs under latency budgets.
• Data curation and filtering, including the sort of data-selection and data-synthesis workflows the paper references for latent reasoning.
• Automated response revision, where the interpreter-restored feedback produced 57.50% Full and 90.12% criterion satisfaction on the frozen-reviser test, and the untrained latent-only interface already improved Repair by 9.25 points over no feedback.

Industry relevance. The reported 6.11–6.95 times vLLM speedup at vote@5 and 6.9–7.8 times at vote@9 translate directly into serving and training throughput, and the authors note that latent reasoning introduces no additional per-step computational cost in the equal-length control (1.017 times). The requirement that compressed judgments remain inspectable is also an operational one: the interpreter gives an offline textual view without lengthening the online decision path.

Future Directions

• Better compression strategies. The authors state that Fixed-Rate Chunking and the current heuristic Semantic Chunking only approximate evaluation structure, through fixed token intervals and linguistic boundary cues respectively, and that compression that better preserves the evidence most relevant to the final judgment remains an important direction.

• Faithful retrospective interpretation. Because the interpreter receives renormalized top-K soft labels that preserve only a subset of the vocabulary distribution, the authors call for richer trajectory representations and reconstruction objectives to strengthen the link between recovered criterion states and their supporting evidence.

• Improving individual latent trajectories. The gains from vote@1 to vote@5 motivate reinforcement learning over latent trajectories, building on approaches such as Latent-GRPO.

• Cheap repeated evaluation at scale. The paper frames shorter generation and lower inference time as enabling more affordable repeated reward evaluation in policy learning, data selection, and related workflows.

Target Audience

Researchers and engineers working on reward modeling, LLM-as-judge systems, and inference efficiency for large language models, particularly those already familiar with rubric-based evaluation or with latent reasoning methods such as Latent-SFT and Latent-GRPO. It is also relevant to practitioners who deploy generative judges in high-volume policy optimization or response-selection loops and need to weigh accuracy against serving cost, and to interpretability researchers interested in the criterion-level intervention and reconstruction protocols used here.

Authors’ abstract

Reward modeling often requires jointly representing and reasoning over multiple evaluation criteria, yet verbalizing this process token by token can incur substantial inference cost. Recent work on latent reasoning suggests that continuous states may support this computation more compactly. We introduce LatentGRM, a latent evaluation framework built on semantic chunking, compression, and reconstruction. By using the structure of rubric-guided evaluations to guide compression, LatentGRM learns compact continuous trajectories that support autonomous pairwise judgments without generating textual assessments. A separate interpreter reconstructs evaluation text from these trajectories, providing an offline view of the information retained under compression. Under matched training data and backbones, LatentGRM achieves competitive aggregate preference accuracy relative to explicit Supervised Fine-Tuning (SFT) judges at both 4B and 8B scales. Across four benchmark domains, LatentGRM-8B compresses evaluation trajectories by 8.9--9.2x and reduces total judge inference time by 6.1--7.0x at vote@5. Controlled rubric interventions show that criterion-dependent preference information is carried through the latent sequence. Together, these results demonstrate that continuous latent evaluation can substantially reduce inference cost while preserving competitive judgment quality.