RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models

AI summary

Overview

Research area: Trustworthy AI and interpretability of Large Reasoning Models (LRMs) — specifically, whether a model's written chain of thought actually drives its answer.

Technical level: Intermediate. The paper includes formal definitions (five numbered definitions in Section 2), but the central idea — injecting a flawed chain of thought and checking whether the model's answer follows it — is intuitive and accessible.

Scope: The paper formalizes "reasoning faithfulness" as two testable conditions, builds a 7,186-instance benchmark (RFEval) over seven tasks, and evaluates twelve open-source LRMs under controlled output-level counterfactual interventions.

What This Paper Is About

Large Reasoning Models produce step-by-step rationales, but a rationale that sounds convincing may not be the actual reason the model gave its answer. The authors argue that measuring accuracy alone cannot detect this gap, and they build a benchmark that tests whether stated reasoning is both internally coherent and causally responsible for the final answer. Their goal is to separate "the model got it right" from "the model's reasoning is genuinely what produced the answer."

Key Contributions

1. A formal definition of reasoning faithfulness built from two testable conditions: stance consistency (the output forms one coherent argumentative chain from reasoning to explanation to answer) and causal influence (the stated reasoning causally shifts the model's stance or answer under intervention), explicitly decoupled from accuracy.
2. RFEval, a benchmark of 7,186 instances across seven tasks — Code Generation, Mathematical Reasoning, Logical Reasoning, Table Reasoning, Context Understanding, Legal Decision, and Paper Review — constructed around controlled output-level counterfactual interventions, with each instance paired with a generated flawed reasoning trace.
3. A large-scale empirical study of twelve open-source LRMs finding unfaithfulness in 49.7% of outputs, overwhelmingly driven by stance inconsistency rather than causal breakdown, with systematic variation across task type and post-training regime.
4. Conceptual and empirical evidence that accuracy is a poor proxy for faithfulness, including within-family ablations and an analysis of RLVR-style rewards suggesting that adding RL on top of supervised fine-tuning can reduce faithfulness even when accuracy holds.

Main Findings

• Unfaithfulness is pervasive: across twelve evaluated LRMs, 49.7% of evaluated instances were unfaithful (reported as 49.73% in the results text), predominantly due to stance inconsistency rather than a failure of causal influence.
• Scores span a wide range across models: overall contrast-conditional RF runs from 73.29% (Qwen3-32B) and 68.52% (LN-Super_v1) at the top to 32.11% (gpt-oss-20b) and 27.50% (gpt-oss-120b) at the bottom. The overall average across models is 50.27% RF with coverage 0.73.
• Scale is not a reliable predictor. Within the Qwen family, going from 8B to 32B raises RF from 41.95% to 73.29%. But the gpt-oss series declines from 32.11% (20B) to 27.50% (120B).
• Brittle, convergent tasks fail most. Lowest task averages are Code Generation (24.18%) and Mathematical Reasoning (28.06%); higher scores appear in Legal Decision (70.17%), Logical Reasoning (58.28%), Paper Review (58.03%), and Table Reasoning (57.92%), with Context Understanding at 51.66%. The authors attribute this to convergent tasks forcing local corrections, producing more "silent corrections."
• The failure location differs by model family. gpt-oss and Magistral-Small often break early, at the r′ → r_new handoff (difficulty responding coherently to a flawed premise); Qwen and R1 families more often break late, at r_new → e′ or r_new → a′ (a disconnect between updated stance and final exposition).
• Post-training regime matters more than scale. In within-family ablations, moving from Base to SFT-only largely preserves RF (MiMo-7B: 59.33% → 60.05%; Olmo-3-7B: 65.87% → 61.38%), while SFT+RL reduces it (MiMo-7B: 46.32%; Olmo-3-7B: 50.93%) at comparable coverage (0.72 and 0.73).
• RLVR rewards do not reward coherence. Using the Open-R1 reward on the DeepSeek-R1 family, the average reasoning-step reward is 0.6280 for stance-consistent outputs (χ=1) versus 0.6711 for stance-inconsistent outputs (χ=0) overall — slightly higher for incoherent traces, consistent with reward for "reasoning shells."
• Accuracy and faithfulness are statistically decoupled once model and task are controlled. Weighted Pearson r = 0.090 (95% CI [−0.141, 0.312], p ≈ 0.445); Weighted Spearman r = 0.145 (95% CI [−0.086, 0.362], p ≈ 0.216), with n_eff = 74.2.
• Contrast coverage varies by task. Coverage is generally high for Mathematical Reasoning and Context Understanding (median c ≈ 0.9 across models) but uniformly low for Paper Review (most models c ≈ 0.35–0.45), because many baselines already align with the flawed stance and are excluded.
• One outlier is excluded from aggregates. Qwen3-8B's Paper Review score of 3.09% is reported for completeness but excluded because a large fraction of baseline outputs lacked any reasoning segment (empty <think> content), making stance consistency practically impossible to satisfy.

Methodology in Plain English

1. Define faithfulness operationally. The authors avoid trying to inspect a model's internal weights. Instead they define faithfulness purely from text: split each output into reasoning (r), explanation (e), and final answer (a), then check (i) stance consistency — whether the whole sequence forms an unbroken chain of the same stance, or explicitly justifies a change; and (ii) causal influence — whether injecting a contradicting piece of reasoning actually shifts the model's stance or answer.
2. Build paired counterfactual data. For each problem, they prompt OpenAI's o3-2025-04-16 with task-specific prompts and hand-crafted few-shot exemplars to generate a plausible but flawed reasoning trace r′ (subtle fallacy, calculation error, contextual misread) aimed at a specific incorrect stance, plus an explanation of the introduced flaw.
3. Validate the generated reasoning in two stages. First, gpt-5-2025-08-07 screens each generation against four criteria: misleading sufficiency, logical soundness, plausible subtlety, and uniqueness for multiple-choice items. Second, eight graduate students in NLP/ML, trained on the same rubric, independently reviewed 70 randomly selected samples, two reviewers per item, achieving percent agreement P_a = 0.855 and PABAK = 0.710. They started with 8,499 instances and removed 1,313, leaving 7,186.
4. Run baseline and intervened generations. Each model answers the original prompt and then the same prompt with r′ appended after the assistant/thinking tags (e.g., <|Assistant|><think>). All outputs are generated with greedy decoding (temperature 0). Pairs with empty or truncated outputs, missing reasoning or answer, outputs exceeding 32,768 tokens, or failing the contrast precondition (S(r′) = S(r)) are dropped.
5. Score with an LLM evaluator. o3-2025-04-16 extracts stances per component and detects flaw identification. Code Generation answers are scored by public test cases (all pass = "correct," any fail = "incorrect"). A human check on 1,035 component-level decisions gave micro-F1 0.952 (95% CI [0.937, 0.963]) for stance extraction and accuracy 0.938 (95% CI [0.922, 0.951]) for flaw identification.
6. Report contrast-conditional RF. Because only pairs where the injected reasoning opposes the model's own stance give an interpretable test, the paper reports RF on contrast-satisfying pairs, plus contrast coverage c(M) describing how often that condition holds.

Why This Matters

Impact on research. The paper argues that reasoning faithfulness should be reported alongside accuracy, and that the community should treat "accuracy is not a proxy for faithfulness" as an empirically supported claim rather than an intuition. It also reframes RLVR-style reward design as a possible cause of unfaithfulness, not just a training-efficiency choice.

Real-world applications:

• Clinical decision support, where a persuasive but unfaithful rationale could conceal the real driver of a recommendation.
• Legal analysis and legal decision support, one of the task categories the benchmark explicitly covers.
• Human resources and hiring workflows, a domain the authors cite where unfaithful explanations can obscure the influence of protected-attribute bias.
• Debugging and auditing deployed reasoning systems, giving teams a protocol for checking whether a model's stated reasoning actually controls its output.

Industry relevance. Organizations deploying reasoning models in regulated or high-stakes settings need reliability guarantees beyond benchmark accuracy. RFEval offers a model-agnostic, behavior-only protocol that does not require white-box access to weights, and the authors' finding that RL post-training can reduce faithfulness is directly relevant to teams designing training pipelines.

Future Directions

• Reduce dependence on a single LLM evaluator. The authors note LLM-based judging may introduce evaluator bias, and suggest aggregating decisions from multiple diverse evaluators such as an "LM-as-a-jury" as a mitigation.
• Extend beyond behavioral evidence. The paper explicitly frames its findings as behavioral, not access to a model's actual cognition; connecting output-level interventions to representation-level analysis remains open.
• Evaluate closed-source models. Results for proprietary, closed-API models are deferred to Appendix G.2, because response integrity mechanisms (e.g., signature verification) complicate editing reasoning traces, and multi-turn simulation proved less robust.
• Redesign RL rewards for coherence. The reward analysis shows existing objectives reward correct final answers without rewarding stance consistency, raising the question of how to shape rewards that explicitly encourage causal influence and structural integrity.

Target Audience

Researchers working on reasoning models, interpretability, and evaluation methodology; safety and trustworthiness teams auditing deployed systems; and practitioners who use chain-of-thought outputs as explanations for decisions. Readers do not need white-box model access to apply the framework, but they should be comfortable with formal definitions and benchmark tables.

Authors’ abstract

Large Reasoning Models (LRMs) exhibit strong performance, yet often produce rationales that sound plausible but fail to reflect their true decision process, undermining reliability and trust. We introduce a formal framework for reasoning faithfulness, defined by two testable conditions: stance consistency (a coherent stance linking reasoning to answer) and causal influence (the stated reasoning causally drives the answer under output-level interventions), explicitly decoupled from accuracy. To operationalize this, we present RFEval, a benchmark of 7,186 instances across seven tasks that probes faithfulness via controlled, output-level counterfactual interventions. Evaluating twelve open-source LRMs, we find unfaithfulness in 49.7% of outputs, predominantly from stance inconsistency. Failures are concentrated in brittle, convergent domains such as math and code, and correlate more with post-training regimes than with scale: within-family ablations indicate that adding current RL-style objectives on top of supervised fine-tuning can reduce reasoning faithfulness, even when accuracy is maintained. Crucially, accuracy is neither a sufficient nor a reliable proxy for faithfulness: once controlling for model and task, the accuracy-faithfulness link is weak and statistically insignificant. Our work establishes a rigorous methodology for auditing LRM reliability and shows that trustworthy AI requires optimizing not only for correct outcomes but also for the structural integrity of the reasoning process. Our code and dataset can be found at project page: $\href{https://aidaslab.github.io/RFEval/}{https://aidaslab.github.io/RFEval/}$