Native Reasoning Models: Training Language Models to Reason on Unverifiable Data

AI summary

Overview

• Research area: Machine learning, large language model reasoning, reinforcement learning, and verifier-free training.
• Technical level: Advanced. The main ideas are accessible, but the paper assumes familiarity with supervised fine-tuning, reinforcement learning, policy gradients, and language model probability modeling.
• Scope: This paper introduces Native Reasoning Training (NRT), a framework that trains language models to generate their own reasoning traces using only question-answer pairs, without expert-written reasoning demonstrations or external verifiers.

What This Paper Is About

The dominant way to train reasoning models is to first imitate human-written reasoning with supervised fine-tuning (SFT), then refine it with reinforcement learning using verifiable rewards (RLVR). This pipeline requires expensive human reasoning annotations and an external verifier, so it works well mainly in domains like math and coding where answers can be checked automatically. NRT asks whether a model can instead learn to reason from ordinary question-answer pairs alone, by rewarding reasoning traces that increase its own confidence in the correct reference answer.

Key Contributions

1. A unified framework for verifier-free reasoning that treats the reasoning trace as a latent variable and optimizes the marginal likelihood of the reference answer.
2. A principled training method, NRT, that requires only question-answer pairs and eliminates the need for expert-written reasoning traces and external verifiers.
3. New intrinsic reward shaping schemes derived from the framework, including geometric mean and weighted-sum variants such as NRT-WS, which target tokens where the model is most uncertain.
4. Empirical evidence that NRT achieves state-of-the-art performance among verifier-free methods, with strong gains in complex reasoning and high robustness to policy collapse compared with prior verifier-free RL methods.

Main Findings

• Verifier-free state of the art: NRT variants achieve the highest average scores among verifier-free methods on Llama-3.2-3B, Mistral-7B-v0.3, and Llama-3.1-8B trained on the same 200K question-answer subset of tulu-3-sft-mixture. On Llama-3.1-8B, NRT-WS(-log p) reaches 56.2 average versus 46.0 for SFT and 50.8 for the strongest prior baseline, RLPR.

• Large gains on complex reasoning: Improvements are especially strong on reasoning-intensive benchmarks. On GSM8K with Llama-3.1-8B, NRT-WS(-log p) raises the score from 29.0 for SFT to 76.0, surpassing RLPR's 65.0. On BBH, NRT-GM improves the 8B model to 54.3, a 13.1-point gain over RLPR.

• Reward shaping is the key lever: Prior verifier-free methods correspond to different aggregation functions in NRT's framework. JLB resembles sequence log-probability, Verifree resembles sequence probability, and RLPR resembles arithmetic mean. Arithmetic mean can reward easy tokens and trivial traces, while geometric mean and weighted-sum schemes are more robust. Weighted-sum schemes that upweight difficult tokens deliver the best overall results.

• NRT resists policy collapse: The RLPR baseline rapidly collapses during training, producing short, low-entropy, repetitive reasoning with low semantic quality. NRT variants maintain high entropy, longer reasoning traces, and stable or improved semantic quality throughout training. This indicates NRT avoids the mode collapse common in some self-rewarding or verifier-free RL approaches.

• Targeted uncertainty reduction works as designed: Token-level analysis shows that NRT-WS variants specifically improve prediction of high-entropy tokens, the tokens the SFT baseline finds hardest. NRT-WS(1/p) increases probability on the most uncertain tokens by up to 15%, while RLPR and NRT-GM show little or no gain on those tokens.

• Small-model robustness: On the 3B model, prior verifier-free RL methods such as JLB, Verifree, and RLPR fail to surpass the SFT baseline. NRT-WS(-log p) achieves 39.9 average versus 36.4 for SFT, showing NRT is more reliable when model capacity is limited.

Methodology in Plain English

NRT starts with a base language model and a dataset of ordinary question-answer pairs. For each question, the model samples multiple candidate reasoning traces, then tries to predict the reference answer after each trace. Each trace receives a reward based on how much it increases the model's probability of generating the correct answer tokens.

The reasoning trace is treated as a hidden variable: it is not directly supervised, but it is rewarded if it helps the model become more confident in the correct answer. The training objective is optimized with reinforcement learning, using group-relative advantages inspired by GRPO. For each question, candidate traces are compared against each other, clipped below a baseline trace, and normalized to reduce gradient variance.

The choice of reward aggregation function is central. Simple arithmetic mean can be dominated by easy tokens and may reward empty or trivial reasoning. Geometric mean forces the model to do well across all answer tokens, because one low-probability token collapses the reward. Weighted-sum schemes go further by assigning higher weight to difficult tokens, measured by the model's baseline uncertainty. This creates a self-reinforcing loop: the model learns to generate reasoning that resolves its own points of confusion.

A small format-supervision loss encourages the model to separate the reasoning trace from the final answer using start-of-reasoning and end-of-thought tokens. This keeps outputs structurally clean without heavily penalizing exploration.

Why This Matters

• Research impact: NRT removes two major dependencies of the SFT+RLVR paradigm: expert-written reasoning demonstrations and external verifiers. It provides a unified way to analyze prior verifier-free methods and their failure modes, such as policy collapse. This expands reinforcement learning for reasoning into domains where correctness is not automatically checkable.

• Real-world applications:

  • Open-ended question answering and customer support, where reference answers exist but correctness is not programmatically verifiable.
  • Summarization and report generation, where reference summaries can guide training without a formal verifier.
  • Educational tutoring and feedback systems, where reasoning quality matters but no unit test or exact-match checker is available.
  • Domain-specific expert QA in medicine, law, finance, or science, where curated question-answer pairs exist but automated verification is difficult.

• Industry relevance: Companies can train reasoning models on proprietary question-answer data without labeling chain-of-thought rationales or building reward models and verifiers. This lowers data-annotation costs, reduces dependence on human cognitive biases, and makes reasoning training feasible in broader commercial domains.

Future Directions

• Designing adaptive or fully learned reward aggregation functions, instead of relying only on handcrafted schemes like geometric mean and weighted sum.
• Improving sample efficiency, since NRT's sampling-based RL approach trades computation for performance and applicability in verifier-free domains.
• Applying NRT principles during pretraining, which could instill foundational reasoning abilities earlier and reduce the need for later fine-tuning.
• Extending NRT to larger models, multimodal reasoning, and tasks where even a reference answer may be unavailable or subjective.
• Developing stronger theoretical guarantees for why certain reward aggregation functions avoid policy collapse and preserve reasoning diversity.

Target Audience

This paper is most useful for machine learning researchers and engineers working on large language model reasoning, reinforcement learning, and post-training. It also benefits practitioners who want to train reasoning models on domain-specific question-answer data without expert-written reasoning traces or external verifiers. Graduate students and technically advanced readers interested in verifier-free RL and reward shaping will find the framework and empirical analysis especially relevant.

Authors’ abstract

The prevailing paradigm for training large reasoning models--combining Supervised Fine-Tuning (SFT) with Reinforcement Learning with Verifiable Rewards (RLVR)--is fundamentally constrained by its reliance on high-quality, human-annotated reasoning data and external verifiers. This dependency incurs significant data-collection costs, risks embedding human cognitive biases, and confines the reinforcement learning stage to objectively assessable domains like mathematics and coding, leaving a wide range of unverifiable tasks beyond its scope. To overcome these limitations, we introduce NRT (Native Reasoning Training), a novel framework that cultivates complex reasoning by having the model generate its own reasoning traces using only standard question-answer pairs, thereby obviating the need for expert-written demonstrations. NRT reframes the training problem by treating the reasoning process as a latent variable. It employs a unified training objective that models reasoning as an optimization problem, intrinsically rewarding paths that increase the model's likelihood of producing the ground-truth answer. This unified perspective allows us to analyze intrinsic failure modes of prior methods, such as policy collapse, and systematically design more robust reward aggregation functions, creating a self-reinforcing feedback loop where the model learns to think in ways that resolve its own uncertainty. Empirical evaluation on Llama and Mistral model families demonstrates that NRT achieves state-of-the-art performance among verifier-free methods, significantly outperforming standard SFT baselines and prior verifier-free RL methods. Our approach yields particularly strong performance gains in complex reasoning domains and exhibits high robustness to policy collapse, offering a general, scalable path toward building more powerful and broadly applicable reasoning systems.