Inherited, Not Invented: Tracing the Provenance of Wait in Reasoning Models

AI summary

Overview

Research area: Machine Learning — the training dynamics and provenance of reasoning behaviour in large language models, sitting at the intersection of reinforcement learning for reasoning (RLVR), training-data analysis, and computational linguistics.

Technical level: Intermediate. The core ideas are accessible, but readers benefit from familiarity with post-training pipelines (pretraining, mid-training, supervised fine-tuning, preference tuning, RL with verifiable rewards) and with basic signal-detection and bootstrap statistics.

Scope (one sentence): The paper traces where the word “wait” actually comes from across the released training histories of open reasoning models, and measures what each training stage — not just RL — does to its frequency, form, position, usage, and evidential value.

What This Paper Is About

Reinforcement learning is widely credited with teaching reasoning models to write “wait” before reconsidering a step, an effect often described as an “aha moment.” The authors ask a narrower, testable question: does RL actually invent this marker, or does the model inherit it from its training data, with RL only reshaping something already present? They answer by treating successive model checkpoints as a historical corpus and reading the word the way a historical linguist would — from its occurrences alone, without labels, judges, or access to model internals.

Key Contributions

1. A provenance test for a reasoning marker. By tracing “wait” through pretraining and mid-training checkpoints, RL trajectories, zero-RL runs from bases that lack it, and distill/fine-tuned descendants, the authors show that in all traced lineages but one the marker originates in pre-RL training data rather than being coined by RL.
2. A recipe-by-recipe account of what RL does to an inherited marker. Both released RL-Zero recipes examined (OLMo-3, MiMo) multiply the marker, whereas the authors' own KL-free GRPO from the same OLMo-3 base does not; RL after supervised fine-tuning changes the rate little; and their GRPO recipe erases a marker that fine-tuning had seeded.
3. A label-free measurement instrument. Frequency, sentence-initial position, orthographic form, chaining, erosion, and usage-drift indices, plus an injected-error “alarm” test yielding signal-detection statistics (d′ and c), all computed from sampled outputs. Applied unchanged to 120 model endpoints and 96 checkpoints.
4. Reusable resources. A provenance map of 120 model endpoints with lineages and marker indices, plus blind revision audits with expert-consensus labels, released under CC-BY-4.0.

Main Findings

• Zero-RL almost never coins the marker. Across 18 zero-RL runs from 9 Qwen2.5, Mistral and Qwen3 bases (0.5B–32B), accuracy rose by a median of 42.9 points while “wait” stayed near absent; in 17 of the 18 runs the exact 95% upper bound stayed below 0.06 per 1k tokens, and the registered near-absence threshold is 0.05 per 1k. Only 1 of 18 runs exceeded it.
• Large accuracy gains do not require the word. Open-Reasoner-Zero-32B moves MATH-500 from 10.8% to 91.4% while writing “wait” 4 times in 2,379,766 sampled tokens (base: 0 in 424,454). The authors' GRPO on Qwen2.5-Math moves 16.9% to 76.0% at 7B and to 74.4% and 74.2% at 1.5B, with 0 occurrences at the final checkpoints.
• The marker arrives with training data. In OLMo-3, “wait” is at or below 0.04 per 1k during pretraining and rises to 4.03 per 1k (88.6% sentence-initial) after mid-training. Separately, 13 of 14 models fine-tuned on reasoning traces from a non-marking base adopt the marker at ≥0.5 per 1k (median 3.54 per 1k).
• One documented exception. General-Reasoner-Qwen3-4B rises from 71.2% to 79.0% on MATH-500 and writes the marker 0.29 per 1k (95% CI [0.24, 0.34]) from a base with 1 occurrence in 434,190 tokens; the same recipe on Qwen3-14B-Base stays below the threshold at 0.04 per 1k. The authors hypothesize the marker was latent in that base.
• Released RL-Zero recipes multiply an inherited marker. OLMo-3 RL-Zero rises from 4.9 per 1k to a peak of 17.0 at step 1900 and ends at 12.4 at the released checkpoint (12.68 ± 1.55 over six decoding seeds; Holm-adjusted p < 0.001). MiMo-7B RL-Zero rises from 4.95 to 6.62. The same rise appears on other benchmarks, e.g. GSM8K 4.62 to 17.10 per 1k.
• A different recipe does not. Run 400 steps from the same OLMo-3 base under identical evaluation, the authors' KL-free GRPO ends at 0.81 times the base rate on all responses and 0.93 on finished ones, versus 1.58 for the released run, with disjoint intervals at steps 200 and 400.
• The inherited marker's form wears down under RL-Zero. The share written “Wait,” falls from 88.9% to 37.5%, reduplicated chains (“Wait, wait”) rise from 117.6 to 213.0 per 1k occurrences, and marker-initial sentences shrink from 11.8 to 6.1 words. Position does not move: 89.6% sentence-initial at the base, 89.4% at the end. Usage distribution shifts with Jensen–Shannon divergence 0.05 against a noise floor of 0.002.
• RL after fine-tuning behaves differently. Along OLMo-3 Think the rate changes little (3.86 to 4.52 per 1k) and the comma form is restored (55.0% to 78.2%, Holm-adjusted p = 0.029); OLMo-3.1-32B-Think stays flat (4.52 to 4.23). The SFT trajectory of the same base barely moves (rate 5.1 to 4.8; comma share 92.1% to 91.7%).
• RL can erase a seeded marker. Fine-tuning Qwen2.5-Math-1.5B on 512 correct traces whose sentences open with “Wait,” yields 7.52 per 1k; after 400 GRPO steps there are 0 occurrences in 3,066,514 tokens, accuracy rises from 41.0% to 77.6%, and responses shrink from 2,466 to 767 tokens. A registered control with an 8,192-token limit rules out length as the cause (7.45 to <0.01 per 1k among finished responses; A = 0.0003, 95% CI [0.0001, 0.0006], against a registered threshold of 0.2).
• Under RL-Zero the marker becomes a more sensitive but more liberal alarm. For any revision marker, d′ rises from 0.57 to 1.09 (95% CI [0.97, 1.24]) and the criterion c falls from 0.11 to −0.24. For “wait” alone, d′ rises from 0.77 to 1.29 (CI [1.16, 1.44]) and c falls from 0.46 to 0.01; it follows 73.7% of error-injected prefixes (base 47.1%) but also 25.4% of error-free ones (base 19.9%). This contradicts the authors' preregistered prediction that the marker would bleach.
• An LLM judge can reverse the human trend. On a 60-sentence blind audit with expert-consensus labels, the primary judge agrees at κ = 0.64 yet reports a revision rate falling from 26.0% to 12.9% along RL-Zero, while the consensus rate rises from 1.3% to 15.9%. A second judge tracks consensus (1.5% to 12.9%, κ = 0.71).
• Marker frequency is not evidence of RL-induced reasoning. Pre-trained bases whose mixtures include reasoning data already write the marker (MiMo-7B-Base 4.95 per 1k, SmolLM3-3B-Base 4.18 per 1k), and all four Qwen3.5 bases write it (0.93 per 1k at 9B to 1.96 per 1k at 0.8B).
• Marker forms track the traces. OpenReasoning-Nemotron-7B opens sentences with “but” (98.3 per 1k) and “perhaps” (28.1 per 1k) instead of “wait” (0.2 per 1k), and no other reflection opener in the authors' inventory substitutes for “wait” in the zero-RL runs.

Methodology in Plain English

The authors exploit unusually open model releases to read a word's history the way a linguist reads a historical text.

• Treat checkpoints as a diachronic corpus. They use three kinds of lineage: fully open OLMo-3, with pretraining, mid-training, base, and both RL-Zero and SFT/DPO/RL trajectories (plus a 32B replication); zero-RL runs from bases that do not use the marker (their own GRPO on Qwen2.5-Math at 1.5B with two seeds and 7B, plus DAPO and Open-Reasoner-Zero on Qwen2.5-32B); and fine-tuned descendants trained on reasoning traces. They evaluate MATH-500 with fixed decoding per series, sampling 2–8 responses per problem, and also check AIME24, AIME25, AMC23, GSM8K and MMLU-Pro.
• Match occurrences by spelling variant. All forms (“Wait,”, “wait”, “Wait.”, and so on) are matched on detokenised text, and frequency is occurrences per 1k generated tokens.
• Build label-free indices from the occurrences only. Sentence-initial share and positional entropy for fixation; the “Wait,” share for form; the share continuing a reduplicated chain for bonding; token distance to the next mathematical action and sentence length for erosion; and Jensen–Shannon divergence of usage clusters against the base's split-half noise floor for usage drift.
• Use an injected-error alarm test. For each problem they build a prefix of correct steps and a copy with one injected arithmetic error, continue both, and record whether a revision marker appears within two sentences. Hit rate and false-alarm rate are converted into d′ (sensitivity) and c (liberality), following standard signal-detection theory.
• Keep statistics honest. Exact Poisson intervals for rare counts, a cluster bootstrap over problems for ratios and for d′, Spearman correlations against RL step with Holm correction, and no pooling across series. The authors note that each open checkpoint series is a single training run, so intervals cover evaluation sampling variance rather than training variance.
• Validate against human labels. Three blind audits use consensus labels from a panel of ten annotators with PhD-level training in related fields; inter-annotator agreement is not reported, and the audits are described as descriptive rather than load-bearing.

Why This Matters

The paper reframes a headline claim in the reasoning-model literature: a rise in “wait” during RL does not show that RL created reflection, because the starting checkpoint may already write it. Uncertainty heuristics that read “wait” as doubt, and chain-of-thought monitors that process traces full of such markers, currently treat the word as a fixed signal. This work shows its evidential weight is a property of the specific checkpoint, and offers a cheap way to calibrate.

Real-world applications:

• Monitor calibration: anyone deploying a chain-of-thought or uncertainty monitor based on reflection words should recalibrate on the exact checkpoint being read, since RL-Zero changes both sensitivity and liberality.
• Training-data curation: teams that want reasoning markers must decide whether to seed them through mid-training or trace fine-tuning, since zero-RL from a marker-free base will not produce them.
• Model auditing and documentation: the provenance map format — endpoint, lineage, recipe group, marker indices — is a template for reporting where a behaviour came from.
• Evaluation quality control: the judge audit is a concrete warning that LLM-as-a-judge revision metrics can invert a human-labelled trend, relevant to anyone scoring reflective behaviour at scale.

Industry relevance: the findings bear directly on post-training recipe choices (RL-Zero versus GRPO versus SFT-then-RL), on claims made in model release notes about emergent reflection, and on the reliability of automated behavioural evaluation used in compliance and safety reviews.

Future Directions

• Separate the roles of data source and RL recipe. The authors state explicitly that the separate roles of the inherited source and the specific recipe remain open, since compared lineages differ in data and recipe alongside the marker.
• Extend beyond the tested regime. All RL runs involve 0.5B–32B models trained with outcome rewards on English mathematics; frontier-scale RL, other domains, other languages, and process rewards remain untested.
• Look past lexical form. Measurements capture the written marker, not underlying reflective behaviour; a model can reflect without writing “wait,” and markers outside the inventory escape detection. Probing internal states alongside output-level indices is a natural next step.
• Improve behavioural evaluation. Judge-based quantities agree only weakly to moderately with human consensus, and the audits use consensus labels on a sample with no independent double annotation, so inter-annotator agreement is not reported. Independent double annotation and better-validated judges are needed.
• Broaden the alarm test. The injected-error test covers only the arithmetic errors the authors inject, and several preregistered predictions did not hold, leaving room for richer error taxonomies.

Target Audience

Researchers and engineers working on reasoning-model post-training, RL with verifiable rewards, and chain-of-thought monitoring will get the most from this paper, along with evaluation specialists who use LLM-as-a-judge for behavioural labelling. It is also useful for computational linguists interested in applying grammaticalization and lexical-semantic-change methods to model checkpoints, and for policy or safety reviewers who need to judge claims that a model “learned to reflect” during RL. Readers wanting a first pass can take the four headline results — zero-RL rarely coins the marker, training data supplies it, recipes differ in what they do to it, and marker frequency alone proves nothing — without engaging the statistical apparatus.

Authors’ abstract

Reinforcement learning (RL) is credited with teaching reasoning models to write {wait} before reconsidering a step, yet large RL accuracy gains do not require the word. This view can confuse what RL discovers with what a model inherits from its training data. We trace the word through open models' training histories, reading it with label-free indices of frequency, position, form and usage and an injected-error test, all computed from sampled outputs. We find that zero-RL almost never coins the marker: on bases that lack the word, RL raises MATH-500 accuracy substantially while {wait} stays near absent. It arrives with training data: mid-training in OLMo-3, and fine-tuning on reasoning traces installs it in most models whose base lacks it. Where the base already uses it, both released RL-Zero recipes (OLMo-3, MiMo) multiply it, whereas KL-free GRPO from the same base does not; OLMo-3's recipe wears down its form and makes it a more sensitive but more liberal alarm; RL after supervised fine-tuning barely changes it, and our GRPO recipe erases a marker seeded by fine-tuning. Finally, we show that marker frequency alone does not indicate RL-induced reasoning and caution against LLM-as-a-judge revision metrics, as automated judges can reverse human-labelled trends.