Judging the Justification: Decision Certificate Supervision for Generative Reward Models

AI summary

Overview

Research area: Natural Language Processing / LLM alignment — specifically generative reward models (GenRMs) and LLM-as-a-Judge systems trained for preference evaluation.

Technical level: Intermediate. The paper assumes familiarity with supervised fine-tuning, DPO, and reward modeling, but its central idea (judging the quality of a written justification rather than only the verdict) is explained in accessible terms.

Scope: A reference-free framework, DeCert, that scores generated judging rationales along three dimensions and converts those scores into SFT targets and DPO preference pairs, evaluated on two 8B model families across in-distribution and out-of-distribution benchmarks.

What This Paper Is About

Generative reward models produce a natural-language rationale before picking which of two responses is better. A judge can pick the right winner while giving a bad reason — citing unsupported evidence, ignoring the issue that actually decides the comparison, or even contradicting its own stated analysis. Existing fixes compare generated rationales against human-written or synthesized reference explanations, which is expensive and still does not test whether a rationale actually supports its own verdict. This paper asks what makes a rationale a well-supported justification without any reference explanation, and turns the answer into a training signal.

Key Contributions

1. A reformulation of rationale supervision. The authors treat each judgment rationale as a decision certificate — a comparative argument that must support its verdict — and define quality in terms of input-grounded validity and verdict consistency rather than agreement with a reference explanation.

2. The DeCert framework. Three complementary, reference-free scores are defined: Soundness (are the cited evidence and local inferences valid?), Coverage (does the rationale address the material issues that actually decide the comparison?), and Decision Support (does the verified content actually entail the declared verdict, accounting for unresolved uncertainty?). They are combined by geometric mean into a rationale score.

3. Outcome-controlled preference construction. Certificate quality is combined with verdict correctness so that any correct-verdict judgment outranks any incorrect one, while rationale preference pairs are built among sampled judgments that all reach the correct verdict, isolating justification quality as the only difference between the preferred and dispreferred example.

4. An empirical study across two model families and multiple benchmarks showing gains in preference accuracy, out-of-distribution generalization, rationale quality, sensitivity to changed evidence, and positional robustness.

Main Findings

• In-distribution accuracy improves. On the HelpSteer3 test set (2k examples), DeCert raises original-order accuracy by 9.1 points for Llama-3.1-8B and 6.3 points for Qwen3-8B over the corresponding base judges (p < 0.0001). The full DeCert Llama model reaches 73.0 accuracy and the Qwen model 71.5.

• Gains hold out of distribution. Against the same-data Outcome-only Signal variants, DeCert lifts RM-Bench overall accuracy from 63.3 to 67.0 (Llama) and from 73.5 to 76.2 (Qwen), and JudgeBench positional-consistent accuracy from 39.1 to 42.6 and from 48.3 to 52.0. DeCert-Qwen reaches an average of 64.1 across the two benchmarks, the best overall among the compared open generative judges under these settings.

• All-correct judgments carry real supervision. Adding certificate-ranked rationale preference pairs to outcome-only training raises positional-consistent accuracy from 46.7% to 52.9% on RM-Bench (+6.2 points) and from 39.1% to 42.6% on JudgeBench (+3.5 points) — a gain available only from examples whose verdict labels are identical.

• Rationale quality improves on an external, reference-based metric. On HelpSteer3-Atomic, measured by RationaleRM-style average precision, the overall score rises from 0.2212 without rationale pairs to 0.2624 with them, and the multilingual score from 0.2973 to 0.3589, despite DeCert never using human reference rationales in training.

• Flaw rates drop across all seven audited categories. Non sequiturs fall from 21.5% to 3.3%, generic correctness claims without concrete evidence from 35.0% to 25.1%, and single-sided praise from 3.6% to 1.7%. Generic unsupported claims remain the most common residual flaw.

• Rationales respond to counterfactual edits. On 1k minimally edited pairs, Evidence Adaptation Rate rises from 64.5% to 88.9% and Stale Rationale Rate falls from 34.0% to 11.1% for Llama; for Qwen, EAR rises from 87.2% to 93.4% and SRR falls from 30.5% to 7.3%.

• Positional bias is mitigated without positional training. On HelpSteer3, swap agreement improves by 13.0 points (Llama) and 13.5 points (Qwen), and positional-consistent accuracy by 14.8 and 9.3 points — exceeding the original-order gains — with no swap augmentation, paired-order training, or explicit consistency objective. The headline "up to 14.8 points" refers to this PCA gain.

Methodology in Plain English

Step 1 — Collect a pool of judgments. For each training comparison, the authors sample several complete judgments (rationale plus verdict): 3 from GPT-4o and 5 from Qwen3-8B. Training data comes from HelpSteer3-Preference; roughly 12k instances are sampled, balanced across General, STEM, Code, and Multilingual tasks.

Step 2 — Break each rationale into a claim graph. An extractor parses the rationale into atomic claims with types, associated evaluation criteria, cited evidence, and parent-claim links, forming a directed acyclic graph. The extractor only structures what the rationale already says; it does not invent missing reasons.

Step 3 — Score soundness. A natural language inference verifier gives each claim a grounding score (is the factual content supported by the cited evidence?) and a local entailment score (does the conclusion follow from its parents?). Validity propagates through the graph by taking the minimum over a claim and all its ancestors, so one unsupported premise poisons everything built on it. Per-criterion scores are combined with the policy's normalized criterion weights.

Step 4 — Score coverage. A separate LLM analyzer reads only the request, the two responses, and the evaluation policy — never the sampled rationales or verdicts — and lists at most six material issues, each with a criterion, an importance weight between 1 and 3, and a comparison direction. Claims are then matched one-to-one against these issues with match scores of 0, 0.5, or 1, using maximum-weight matching, so a vague statement cannot collect credit for several distinct issues. Matching weights are multiplied by claim validity, so unsupported claims earn no coverage credit.

Step 5 — Score decision support. For each criterion, verified comparative statements in the rationale narrow an interval of admissible comparison values (an explicit tie pins the interval to {0}; an unaddressed criterion keeps the full range from −1 to 1). The framework then computes the worst-case policy margin over every comparison consistent with those intervals, via a closed form that subtracts total unresolved uncertainty from the signed score at the center of the feasible region. Unaddressed criteria keep their weight as potential counter-evidence rather than being renormalized away. The margin, mapped to [0, 1], is the decision-support score.

Step 6 — Combine and rank. The three scores are combined with a geometric mean so a low score on one dimension cannot be offset by high scores elsewhere. Total score is (outcome correctness + λ × rationale score) / (1 + λ) with 0 < λ < 1, guaranteeing correct verdicts always outrank incorrect ones.

Step 7 — Build training data. Pools mixing correct and incorrect verdicts yield general preference pairs from the highest- and lowest-scoring candidates, and the top candidates also serve as SFT targets. Pools where every candidate is correct yield rationale preference pairs, retained only if the rationale-score gap exceeds a threshold δ; pools with only incorrect verdicts are discarded. The result is about 10k SFT samples and 8k DPO pairs, of which about 3k are rationale preference pairs.

Step 8 — Train in two stages. SFT followed by DPO with Low-Rank Adaptation, on Llama-3.1-8B and Qwen3-8B. Evaluation uses the HelpSteer3 test set (accuracy, swap agreement, and positional-consistent accuracy, PCA), plus RM-Bench and JudgeBench as out-of-distribution benchmarks.

Why This Matters

The paper separates two things that benchmark leaderboards usually conflate: getting the right answer and giving a defensible reason for it. Because DeCert needs no reference explanations, the supervision signal is cheap to produce at scale, and because it targets the reasoning rather than the response-order artifacts, it improves robustness as a side effect.

Real-world applications:

• RLHF reward signals. The paper cites RationaleRM's finding that reward models can score well on static preference benchmarks yet perform poorly as reward providers in RLHF; better-justified judges are more likely to be useful in that role.
• Automated evaluation pipelines. Teams using LLM judges to grade model outputs can audit why a judge preferred one output, catching judges that reward verbosity or style rather than substance.
• High-stakes review settings. The introduction frames unreliable judging as a risk to safe deployment where a stated rationale conflicts with the final output, misleading users.
• Preference data curation. Rationale preference pairs let practitioners extract additional training signal from comparisons where every sampled judgment is already correct — the common case once judges get reasonably good.

Industry relevance: The training recipe uses modest data (8k DPO pairs versus 173K pairwise-equivalent preferences for RewardAnything-8B and 2.54M for GRAM-R²) and 8B base models with LoRA, so the approach is comparatively cheap to reproduce. The result that positional consistency improves without swap augmentation also removes a standard data-doubling trick from the pipeline.

Future Directions

• Reducing dependence on LLM analyzers. The limitations section notes that certificate assessments rely on LLM-based analyzers, so extracted issues may inherit their reliability and biases; more robust or verifier-grounded analyzers are a natural next step.
• Scaling beyond two 8B families. Whether the same gains hold for larger models or for judging settings other than pairwise comparison is explicitly left unstudied.
• Closing the residual flaw gap. Generic unsupported claims remain the most common remaining flaw after DeCert, which the authors identify as an important direction for improvement in evidence grounding.
• Testing faithfulness, not just quality. The analyses measure rationale quality and evidence sensitivity but do not establish that rationales faithfully reflect the model's latent decision process — an open question the authors flag directly.

Target Audience

Researchers and practitioners working on reward modeling, LLM-as-a-Judge systems, and preference optimization who want a training signal that goes beyond verdict labels without paying for reference explanations. It is also useful for evaluation engineers concerned with positional bias and with whether judge rationales can be trusted, and for readers interested in process-level supervision outside the solver-reasoning setting of conventional process reward models.

Authors’ abstract

Generative reward models (GenRMs) generate natural language rationales to justify preference judgments, yet a correct verdict does not guarantee a sound justification. Existing approaches often rely on reference explanations to supervise rationales, but obtaining such references can be costly and does not necessarily establish whether generated rationales support the resulting verdict. To address these limitations, we propose DeCert, a framework that evaluates the validity and verdict consistency of generated rationales without requiring reference explanations. Crucially, DeCert translates these assessments into preference supervision that accounts for both verdict correctness and justification quality. In particular, when multiple sampled judgments share the same correct verdict, differences in certificate quality enable DeCert to construct rationale preference pairs that distinguish stronger from weaker justifications. Extensive empirical studies across two model families and multiple in- and out-of-distribution benchmarks demonstrate that DeCert consistently outperforms strong generative judge baselines while improving rationale quality. Moreover, DeCert effectively mitigates positional bias, improving position-consistent accuracy by up to 14.8 points. These results highlight the importance of supervising not only which verdict is correct, but also how that verdict is justified.