Evaluating the Role of Verifiers in Test-Time Scaling for Legal Reasoning Tasks

AI summary

Overview

• Research area: Natural Language Processing, specifically test-time scaling (TTS) and reward-model-based verification for legal multiple-choice question answering.
• Technical level: Intermediate. Readers should be comfortable with LLM inference concepts such as Chain-of-Thought prompting, sampling, tree search, and reward models (ORMs vs. PRMs).
• Scope in one sentence: An empirical study of whether verifier-guided test-time scaling methods (Best-of-N and Diverse Verifier Tree Search) can beat simple Majority Vote on five legal MCQA benchmarks across three generator sizes and seven reward models.

What This Paper Is About

Test-time scaling improves LLM accuracy by spending more compute at inference time, and verifier-guided methods like Best-of-N and tree search have worked well in formal domains such as mathematics and programming. The authors ask whether those gains carry over to law, an argumentative domain where reasoning is defeasible and multiple analytical paths can be valid, and where the verifiers themselves are usually trained on general-purpose or formal-domain data rather than legal text. Their goal is to measure when, and whether, verifiers actually help legal MCQA under realistic low-N compute budgets.

Key Contributions

1. A comprehensive comparison of Majority Vote, outcome-verified Best-of-N, and process-verified DVTS using open-source models on legal reasoning, showing verifier-based methods rarely outperform simple voting baselines by significant margins.
2. Systematic ablation studies showing that both verifier model size and domain specialization matter, with legal-domain training providing a distinct advantage that becomes most apparent at larger scales.
3. Evidence that the utility of all TTS methods diminishes as generator capability increases, with even sophisticated verification providing minimal gains for stronger generators.
4. An analysis of supervision type showing PRMs outperform similarly sized ORMs, even when PRMs are deployed outside their intended role as outcome verifiers.

Main Findings

• Majority Vote is a hard baseline to beat. Across benchmarks and generator scales, MV remained strong. BoN and DVTS surpassed MV on average across the five benchmarks only with the Llama-3.2-3B-Instruct generator, achieving around 1.4% average improvement. For larger generators, verification provided no benefit or even a decrease.
• Answer-space size changes the picture. On low-cardinality benchmarks (COLIEE Task 4, MBE, LEXam), verifier methods gave marginal or negative gains over MV. On LEXam-32 with 32 possible answers, DVTS achieved a +12.4% relative gain and Best-of-N +10.2% with the 70B generator at N=16.
• Per-benchmark 70B results are mixed. At N=16 with the Llama 70B generator, Best-of-N relative gains versus MV were +0.6 (MBE BAR), -2.1 (SuperGPQA), -1.3 (LEXam), +10.2 (LEXam-32), and -1.4 (COLIEE Task 4); DVTS results were -4.8, -4.7, -0.5, +12.4, and -2.7 respectively.
• Diminishing returns as generators get stronger. Relative improvement over MV at N=16: BoN + VersaPRM 8B gave +2.94 (3B), +0.16 (8B), +1.56 (70B); BoN + Legal ORM 70B gave +4.46, +2.22, +1.20; DVTS + QwenPRM 72B gave +4.00, +2.00, -0.06.
• Legal specialization helps, mostly at scale. Best-of-N with the authors' reward models (8B and 70B) matched or outperformed general-domain verifiers. The 8B model showed minimal advantages over general verifiers, while the 70B model consistently delivered superior results across numerous benchmarks. The advantage from specialization was most pronounced at the 70B scale.
• Multi-domain PRMs beat out-of-domain ones at matched size. Direct comparison between VersaPRM 8B and QwenPRM 7B showed VersaPRM consistently delivered superior performance in DVTS.
• Process supervision generalizes to outcome verification. PRMs provided consistent benefits as BoN scorers and within DVTS, with improvements concentrated on smaller generators. The authors' 70B model could still perform better than VersaPRM in process supervision despite receiving no process training.
• Hyperparameter sensitivity. Expansion width tuning on BAR with Llama-3.1-8B-Instruct and VersaPRM (N=16) gave 55.6% (width 8, 2 subtrees), 59.5% (width 4, 4 subtrees), and 61.8% (width 2, 8 subtrees); all experiments therefore used width W=2 with T = N/W. For aggregation strategy on BAR with Llama-3.1-8B-Instruct at N=16, VersaPRM favored Mean (62.7%) over Min (62.2%) and Last (59.5%), while Qwen2.5-Math-PRM-72B favored Last (60.5%) over Prod (59.2%), Mean (57.7%), and Min (57.6%).

Methodology in Plain English

The authors test three generators from the Llama family (3B, 8B, and 70B Instruct variants) with Chain-of-Thought prompting and sampling temperature T=0.8, and compare three ways of producing an answer:

• Majority Vote (MV): sample k CoT responses and pick the most frequent answer.
• Best-of-N (BoN): sample N CoT responses, score each with an Outcome Reward Model that gives one score per complete output, and pick the highest-scoring one.
• DVTS: a tree search guided by a Process Reward Model that scores partial reasoning steps.

They use seven reward models spanning three groups: their own legal-plus-general ORMs at 8B and 70B, Skywork-Reward ORMs at 8B and 27B (general), VersaPRM 8B (multi-domain PRM, described as the first open-source multi-domain PRM), and Qwen2.5-Math-PRM at 7B and 72B (math). Evaluation covers five legal benchmarks with different numbers of answer options: COLIEE Task 4 (binary), MBE BAR Exam and LEXam (four options), SuperGPQA Law subset (eight options), and LEXam-32 (thirty-two options). MBE BAR Exam is the only restricted-access benchmark. Three research questions structure the study: value of verification under matched compute (RQ1), impact of domain specialization and verifier size (RQ2), and role transfer between PRMs and ORMs (RQ3). The authors note that generation dominates cost, scaling as Θ(P_M N T²) without KV caching, while BoN adds Θ(P_R N T) and DVTS about Θ(P_R N T (s+1)/2), so all three methods have comparable runtime at a fixed N, with average CoT length of 1000 tokens and s ≈ 10.

Why This Matters

Impact on research. Prior work on verifier-guided TTS concentrated on formal domains such as math and programming; this paper provides a systematic empirical check in an argumentative domain and finds the benefits do not straightforwardly transfer. It also raises a practical warning that strong reward-model benchmark scores (Skywork-RM-27B was selected for its RewardBench performance) do not guarantee better legal QA outcomes, and that gains shrink as generators improve.

Real-world applications:

• Selecting between simple self-consistency voting and expensive verifier pipelines when deploying legal question-answering systems under latency and compute budgets.
• Bar-exam and professional legal assessment tools where answer spaces vary from two options to thirty-two options.
• Legal research assistants that must rank among many candidate answers generated by an LLM.
• Building domain-specialized reward models for regulated professional domains where general-purpose verifiers may not capture domain nuance.

Industry relevance. The finding that defensive, general-purpose approaches like MV often suffice for small answer spaces, while verifier overhead becomes justifiable only as the answer space grows, directly informs inference-compute budgeting. The paper also indicates that investing in high-quality, in-domain reward models is the more promising direction than scaling general verifiers, which matters for organizations deciding where to spend training data and compute.

Future Directions

• Extend the evaluation beyond legal MCQA to other legal tasks such as summarization and open-ended QA, where verification is arguably more complex; the authors explicitly flag open QA as an area warranting further investigation of the MV-versus-verifier relationship.
• Test other model families beyond Llama 3.1 and Llama 3.2, since other architectures might exhibit different improvements from verification.
• Expand the verifier pool with more recent reward models, such as Skywork-v2 Reward Models, and evaluate newer generators like Qwen3 models.
• Address reproducibility limits, since some verifiers used in the study rely on restricted-access data, which constrains full reproduction of certain results.

Target Audience

Practitioners and researchers working on legal NLP, inference-time scaling, and reward-model design; machine learning engineers deciding whether verifier-guided decoding is worth its compute for domain-specific question answering; and applied researchers studying how domain specialization and model scale interact in reward models. Readers without a background in TTS methods will need some grounding in Chain-of-Thought sampling, Best-of-N, and tree search to follow the experimental design.

Authors’ abstract

Test-time scaling (TTS) techniques can improve the performance of large language models (LLMs) at the expense of additional computation and latency. While TTS has proven effective in formal domains such as mathematics and programming, its value in argumentative domains such as law remains underexplored. We present an empirical study of verifier-based TTS methods for legal multiple-choice QA (MCQA) across five benchmarks. Using a family of 7 reward models, we evaluate both outcome-level (Best-of-$N$) and process-level (tree search) verification under realistic low-$N$ budgets. Our analysis systematically investigates how verifier utility is affected by key properties such as domain specialization, model size, and supervision type (process-supervised PRMs vs. outcome-only ORMs), even when applied across different roles.