Base Models Can Reason By Taking a Cue From Training Data

AI summary

Overview

Research area: Machine learning / large language model interpretability, reasoning, and post-training (reinforcement learning).

Technical level: Advanced. The core idea is intuitive, but the paper assumes familiarity with base models versus post-trained models, RL-Zero-style GRPO training, prefilling, pass@1 evaluation, and hidden-state similarity analysis.

Scope: The paper shows that fixing the first one to two tokens of a base model's response can elicit reasoning at the level of its RL-trained counterpart, and that these token cues are learned associations with training data that can be created, redirected, or removed by editing that data.

What This Paper Is About

The paper asks how little intervention is needed to make a base language model reason. The authors find that prefilling just the first two tokens of a response — tokens that contain no instruction or demonstration of reasoning — can raise a base model's math and coding accuracy to the level of a model trained with reinforcement learning, and they then trace those token cues back to associations in the training data.

Key Contributions

1. Token cues match RL. The authors introduce a label-free method (beam search over likely openings, then selection by answer agreement) that finds short fixed openings bringing base models Olmo-3-7B and Qwen3-14B to RL-level accuracy on math benchmarks.
2. RL mostly makes the cue more likely. They show that RL concentrates policy change in the first two output positions and raises the probability of exactly the openings that already elicit reasoning, so prefilling the cue before RL recovers much of RL's accuracy gain.
3. Causal training-data edits create and redirect cues. By editing Olmo-3-7B's mid-training data and retraining, they turn an arbitrary word ("chicken") into an effective reasoning cue, remove an existing cue's benefit by pairing it with question text, and make the prompt instruction "Think duck duck goose" as effective as "Think step by step."
4. Cues also shape safety behavior. As a case study, different openings produce qualitatively different refusal and compliance behavior on safety prompts, with hidden-state similarities that correspond to different training sources.

Main Findings

• A two-token opening can rival RL. Prefilling . \n\n Okay raises Olmo-3-7B's MATH-500 pass@1 from 42% to 78%, compared with 75% after RL. Prefilling ␣Alright, raises Qwen3-14B's from 72% to 87%, matching its RL counterpart.
• Cues transfer across benchmarks. On Olmo-3-7B the fixed tokens raise pass@1 by 1.4× to 1.9× across MATH-500, GSM8K, AMC 23, AIME 2024, and HumanEval; on Qwen3-14B they raise pass@1 by 1.1× to 3× on the four math benchmarks, while HumanEval stays comparable to the no-cue base model.
• Correct answers already start differently. In Olmo-3-7B's MATH-500 responses, 50% of correct answers begin with the token . \n\n (a period and two newlines), versus 14% of incorrect answers.
• The cue search is label-free. Selecting the opening with the lowest answer entropy over 30 MATH training problems lands within 1.1 percentage points of the highest MATH-500 pass@1 among 500 tested openings.
• RL concentrates its change at the start. KL divergence between base and RL next-token distributions peaks within the first two positions and stays small afterward. RL raises the probability of . \n\n Okay from 0.14 to 0.65 in Olmo-3-7B and of ␣Alright, from 0.04 to 0.58 in Qwen3-14B.
• Prefilling substitutes for RL updates. Prefilling the cue before RL updates matches roughly 100 RL steps on Olmo and 50 on Qwen, with further cue-forced training adding little.
• Gains are not just longer generation. The cue increases mean response length from 4.8k to 6.7k tokens, but some openings generate roughly twice as many tokens as the reasoning cue while scoring below no cue. Truncation shows the cue beats no cue from 1k tokens onward and reaches RL-Zero accuracy at roughly half the token cap.
• Training data contains the association. In Olmo-3-7B's mid-training data, "Okay" opens 83% of synthetic reasoning traces.
• Renaming creates a new cue. Replacing "okay" with "chicken" in the 10B-token mid-training mix raises MATH-500 accuracy with the . \n\n Chicken opening from 2.4% to 37.2%, close to the 38.3% that . \n\n Okay reaches on the same mix; GSM8K accuracy with the cue rises from 2.2% to 60.6%.
• Removing a word does not remove its effect, but redirecting does. After renaming, the probability of Okay following the prompt and . \n\n drops from 0.15 to approximately 5×10⁻⁷, yet . \n\n Okay still elicits reasoning. Pairing "Okay," with question text (the redirect mix) makes the model generate questions instead, and accuracy falls to 0.2%.
• Instructions are also learned associations. Replacing "step by step" with "duck duck goose" in training raises MATH-500 accuracy with "Think duck duck goose" from 6% to 17%, while "Think step by step" accuracy rises from 12% to 15%.
• Cues shift hidden states toward different training sources. Comparing layer-24 hidden states to training-document states by cosine similarity, . \n\n Okay shifts similarity toward synthetic reasoning traces, . To toward expository-math sources, and . Answer toward short question-answer sources. . Problem shifts toward a meta-reasoning group but scores only 6.3% pass@1, showing that similarity to reasoning-related states does not imply successful reasoning.
• Style follows the cue. Across 2,000 responses per opening, checking phrases occur in 97% of . \n\n Okay responses but only 5% of . To responses.
• Position matters. Inserting . Okay at paragraph 1 or 2 yields 69–71% pass@1, versus 23–30% at paragraphs 3–6 and 38% with no cue. Successful insertions sustain the shift toward reasoning traces over the next 128 tokens; later insertions produce only a transient shift.
• Cues shape refusal. On XSTest (250 safe and 200 unsafe prompts), ␣I␣'m␣␣sorry increases refusal of both harmful and harmless requests, ␣Okay, reduces refusal of harmful requests, and . \n\n Okay produces more selective refusal, closer to Olmo's Instruct and Think-SFT variants. The harmful-response rate on unsafe requests is 42.4% with ␣Okay, versus 0.2% with . \n\n Okay, despite both shifting hidden states toward reasoning traces.
• Effectiveness is model-dependent. The authors report no effective cue for Llama-3.1-8B among the openings tested.

Methodology in Plain English

The authors separate two things a model does when answering: choosing how to start its response, and continuing from there. Using prefilling, they supply the opening tokens themselves and let the model generate the rest without changing any weights, isolating the effect of the opening from how likely the model was to produce it.

To find openings without reference answers, they use beam search (width 20, depth two, over 30 MATH training problems) to collect 20 likely openings, sample 16 continuations per opening per problem, and pick the opening whose sampled answers agree most — measured as the lowest answer entropy. They discard openings that produce too many unparseable answers (more than 10 percentage points above the no-cue rate).

They then train base models with GRPO on MATH for 300 steps, both normally and with the opening fixed (cue-forced), to see whether RL makes the cue more likely and whether the cue already supplies the accuracy RL would deliver. To test causation rather than correlation, they edit Olmo-3-7B's released 10B-token mid-training mixture — swapping "okay" for "chicken", and replacing paragraph-initial "Question:" labels with "Okay," — and retrain from the same checkpoint with the same recipe, data order, and seed, with a 100B-token run confirming the same cue effect at higher accuracy.

Finally, they look inside the model: they compare layer-24 hidden states from generated responses against hidden states from training documents, finding the ten nearest neighbors by cosine similarity and measuring which training source those neighbors come from. They repeat the setup in a safety case study, using WildGuard to classify refusal and harmful content on XSTest.

Why This Matters

Impact on research. The work sharpens the question of what reinforcement learning adds to a base model. If two fixed tokens recover most of RL's accuracy gain in these runs, then RL in this setting may largely be increasing the probability of behavior the base model could already produce — supporting the view that RLVR shifts probability toward reasoning trajectories the base model already supports rather than expanding its underlying capability. It also reframes prompt instructions such as "Let's think step by step" as learned data associations as much as semantic content, and shows that the same short opening can affect both reasoning and safety behavior.

Real-world applications.

• Cheap inference-time control: prefilling one or two tokens is far less expensive than an RL training run and can serve as a diagnostic or fallback when a base model underperforms.
• Curating synthetic training data: knowing that "Okay" opens 83% of reasoning traces, and that renaming changes cue behavior, lets data designers deliberately pair useful behaviors with cues, or avoid accidental pairings.
• Safety evaluation and deployment: opening tokens measurably change refusal and harmful-response rates, so interfaces that fix or constrain response openings change safety outcomes.
• Debugging and interpretability: comparing generated hidden states to training documents offers a concrete way to trace a behavior back to the data that produced it.

Industry relevance. The findings are directly relevant to teams building post-training pipelines, since they suggest auditing whether an expensive RL stage is teaching new capabilities or amplifying existing ones, and to teams producing large volumes of synthetic data, since the paper shows such data can introduce unintended cue–behavior associations.

Future Directions

1. Test richer instructions. The authors suggest extending the "duck duck goose" intervention beyond a single phrase to more complex instructions, to disentangle semantic content, surface form, and learned associations.
2. Explain model dependence. No effective cue was found for Llama-3.1-8B among the openings tested, which the authors attribute to differences in the structure of reasoning-related training data; identifying that structure is an open question.
3. Look beyond R1-Zero-style pipelines. The analysis covers RL applied directly to a base model without a supervised fine-tuning stage; complex multi-stage post-training pipelines may introduce capabilities the cues cannot elicit.
4. Map unintended cue–behavior associations. As synthetic data becomes more common in pretraining, the paper argues it will matter to know which cue–behavior associations such data introduces and when they help or harm.

Target Audience

Researchers and engineers working on LLM post-training, reinforcement learning for reasoning, and data-centric model development; interpretability researchers interested in linking behavior to training data; and safety researchers studying refusal and jailbreak behavior. Readers should already be comfortable with base versus post-trained models, prefilling, and standard reasoning benchmarks.

Authors’ abstract

In this paper, we study how training data creates associations between the tokens at the start of a base model's response and the reasoning behavior that follows. First, we demonstrate that fixing particular starting token cues makes a base model's performance competitive with that of its reinforcement learning (RL)-trained counterparts on math and coding. For instance, the cue ".\n\nOkay" raises Olmo-3-7B's MATH-500 pass@1 accuracy from 42% to 78%, while "Alright," raises Qwen3-14B's from 72% to 87%. Second, RL makes these cues more likely, while fixing them recovers much of its performance gain over the base model. Third, we trace the reasoning effects of token cues to the training data. We perform causal data interventions to turn an arbitrary word, such as "chicken", into an effective reasoning cue, or remove an existing cue's effect. A similar edit makes the prompt instruction "Think duck duck goose" as effective as "Think step by step" at eliciting reasoning. We also find that the hidden state representations induced by different cues correlate with different document types from the training set. Finally, we extend our study of token cues with a case study in language model safety, finding that different cues elicit distinct refusal and compliance behaviors that correspond to different types of training data.