ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering

AI summary

Overview

Research area: Natural Language Processing — evaluation of reasoning in Small Language Models (SLMs), specifically commonsense question answering.

Technical level: Intermediate. The paper is readable without deep technical background, but familiarity with chain-of-thought prompting, LLM-as-a-judge evaluation, and Process Reward Models helps.

Scope: The paper introduces ReTraceQA, a human-annotated benchmark of 2,421 step-level reasoning traces from seven SLMs across four commonsense QA datasets, and uses it to show that answer-only evaluation overstates SLM reasoning ability.

What This Paper Is About

Current evaluations of small language models in commonsense reasoning almost exclusively check whether the final answer matches the gold answer, ignoring whether the reasoning that produced it was sound. This means a model can be scored as correct even when its intermediate steps contain hallucinations, misinterpretations, or invalid logic. The paper builds ReTraceQA, a benchmark with step-level error annotations, to measure how much this answer-only practice inflates apparent SLM performance, and to test whether large language models and Process Reward Models can serve as reliable reasoning judges.

Key Contributions

1. ReTraceQA, a new gold benchmark: the first benchmark for evaluating SLMs' reasoning traces in commonsense reasoning, consisting of 2,421 manually annotated traces with step-level error locations and qualitative error categorizations across the CSQA, OBQA, QASC, and StrategyQA datasets.
2. Quantitative evidence of metric inflation: up to 24% of flawed reasoning traces (averaging 17.9% across datasets) still reach the correct final answer, showing that answer-only evaluation overestimates model capabilities.
3. Reference-based evaluation of judges: a comprehensive study of closed and open-source LLMs as judges, showing they can often tell whether a trace is correct as a whole but struggle to localize the exact erroneous step.
4. Reference-free evaluation and cross-domain transfer: an assessment of LLM-as-a-judge models and math-trained PRMs on commonsense reasoning, revealing substantial degradation when transferring PRMs across domains.

Main Findings

• Correct answers, flawed reasoning: Process errors — instances with correct final answers but at least one reasoning error — occur in 16.3% of CSQA instances, 14.7% of OBQA, 16.6% of QASC, and 24.0% of StrategyQA, averaging 17.9%.
• Error types are dominated by factual failures: Across subsets, hallucination accounts for 41.9%–62.5% of errors, reasoning errors for 27.9%–35.4%, and misinterpretation for 9.6%–24.1%.
• Reference-free judging is hard: Average F1 across LLM judges is around 54–56%, while math-trained PRMs average 21.1% F1, with several PRMs scoring far lower (Skywork-PRM-1.5B averages 4.4% F1).
• Reference-based judging is better but still imperfect: Average F1 across judges rises to 60.8%, with o1-mini the strongest judge at 74.4% F1. Models detect trace-level correctness more reliably than they localize the first error.
• Scale is not sufficient: Qwen2.5-72B-Instruct outperforms Mistral-Small-24B-Instruct by +35.5% F1 on average, yet DeepSeek-R1, despite being larger than Qwen2.5-72B-Instruct, underperforms across all datasets.
• Error positions cluster mid-trace: Human-annotated errors most commonly occur at steps 3–4; o1-mini's predictions mirror human patterns on CSQA and QASC but show heavier tails, over-assigning blame to later steps.
• Reasoning-aware evaluation lowers SLM scores: Using o1-mini as a reasoning-aware judge rather than an answer extractor raises accuracy by +5.8 points and error recall by +20.6 points on average (accuracy 81.2 to 87.0, error recall 69.2 to 89.8).
• SLM performance drops substantially: Across seven SLMs and four datasets, reasoning-aware evaluation produces an average 18.6 percentage point drop (68.3% to 49.7%); Qwen2.5-7B-Instruct falls from 81.0% to 67.5%.
• Annotation reliability: Inter-annotator agreement on a 100-instance subset (25 per dataset) reached a Fleiss's kappa of 0.84, described as "almost perfect."

Methodology in Plain English

The authors started from 3,334 original commonsense questions drawn from four datasets: 1,221 from CSQA, 500 from OBQA, 926 from QASC, and 687 from StrategyQA. They used seven instruction-tuned SLMs — Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, and Phi-4-mini-instruct (an SLM being defined as a model with no more than 10 billion parameters) — with zero-shot chain-of-thought prompting to produce step-by-step reasoning traces, giving an initial pool of 23,338 traces. An LLM-based answer extractor (xFinder-llama38it) determined whether each trace reached the correct answer. Sampling then reduced the pool to 2,779 unique instances, balancing correct and incorrect traces, balancing models, and keeping each question unique.

Rather than re-segmenting traces automatically (as is needed in math), the authors found the chain-of-thought outputs already had naturally delineated steps, split at double newlines (falling back to single newlines). Three expert annotators with PhD-level backgrounds — also the paper's authors — then labeled each trace with the earliest erroneous step, or marked it fully correct, and assigned one of three mutually exclusive error categories: Misinterpretation (grounding level), Hallucination (content level), and Reasoning (inference level). Problematic instances were flagged with an Invalid tag and removed, leaving 2,421 clean examples.

Evaluation ran in two settings. In reference-free evaluation, judges received only the question and trace and had to predict the first error step; PRMs were used by extracting step-wise correctness predictions and taking the first step flagged incorrect, with rounding or F1-optimized thresholds on a CSQA validation split. In reference-based evaluation, judges additionally saw the gold answer. Two metrics were used: correct (identifying fully valid traces, human-labeled -1) and error (localizing the first erroneous step, labeled i ≥ 0), combined as their harmonic mean (F1). Finally, the best judge (o1-mini) was applied downstream to seven SLMs across four commonsense datasets, comparing answer-only extraction against full trace validation, with traces generated at temperature 0.7.

Why This Matters

Impact on research: The paper shows that standard accuracy metrics systematically inflate SLM reasoning ability by up to 25%, and that benchmarks built for mathematics (ProcessBench, MR-Ben, MR-GSM8K) leave commonsense reasoning largely unexplored despite requiring fundamentally different capabilities. It provides both a diagnostic resource and an argument that process-level evaluation must extend beyond STEM.

Real-world applications:

• Selecting and comparing small, deployable models for on-device or cost-sensitive question answering, where an inflated benchmark score could lead to a poor deployment choice.
• Generating reliable reward signals for training or Best-of-N selection, since reference-free judges currently cannot robustly reward correct intermediate reasoning.
• Building quality filters and self-verification pipelines for assistant systems that answer commonsense questions.
• Auditing model outputs in knowledge-sensitive settings, where hallucination (41.9%–62.5% of errors) rather than faulty logic is the dominant failure mode.

Industry relevance: Organizations increasingly rely on small models for latency and cost reasons, and on LLM-as-a-judge for automated evaluation. This work quantifies the gap between those judges' ability to spot a bad trace overall versus pinpoint where it goes wrong, which directly affects the reliability of automated QA pipelines. It also shows that specialized PRMs trained on math do not transfer to commonsense, implying that domain-specific evaluation infrastructure is needed rather than assuming general transferability.

Future Directions

• Extending the benchmark to multilingual commonsense reasoning, since the current work covers only English and it is unknown whether error distributions vary across languages.
• Broadening beyond the four datasets to other reasoning modalities such as procedural reasoning and narrative comprehension.
• Training specialized PRMs for commonsense and other non-mathematical domains, given that math-trained PRMs transfer poorly.
• Improving error localization specifically, since judges align reasonably well on overall trace correctness but consistently over-assign blame to later steps, capturing error consequences rather than origins.

Target Audience

Researchers and practitioners working on language model evaluation, commonsense reasoning, and process supervision; engineers building automated evaluation or reward-model pipelines; and anyone choosing small language models for deployment who needs to understand why answer-only leaderboard numbers can overstate real reasoning ability.

Authors’ abstract

While Small Language Models (SLMs) have demonstrated promising performance on an increasingly wide array of commonsense reasoning benchmarks, current evaluation practices rely almost exclusively on the accuracy of their final answers, neglecting the validity of the reasoning processes that lead to those answers. To address this issue, we present ReTraceQA, a novel benchmark that introduces process-level evaluation for commonsense reasoning tasks. Our expert-annotated dataset reveals that in a substantial portion of instances (14-24%), SLMs provide correct final answers despite flawed reasoning processes, suggesting that the capabilities of SLMs are often overestimated by evaluation metrics that focus only on comparing the final answer with the ground truth. Indeed, we show that, when employing strong Large Language Models (LLMs) as automated judges for reasoning-aware evaluation rather than answer-only metrics, SLM performance drops significantly across all models and datasets, with scores decreasing by up to 25%.