When Tokenization is Secretly Output Supervision

AI summary

Overview

Research area: Natural Language Processing — tokenization, autoregressive training objectives, and model interpretability.

Technical level: Intermediate. The core argument is conceptual and stated plainly, but the supporting evidence relies on a 2x2 factorial training design and linear/MLP probing of residual-stream activations.

Scope: The paper argues that tokenizer granularity on the output side is a supervision signal, tests this claim with controlled addition experiments and probing, and documents that the field rarely reports the tokenization regimes it compares across.

What This Paper Is About

Tokenization is normally described as an input preprocessing step: a way of turning text into tokens before a model sees it. This paper argues that framing misses half the picture, because in autoregressive models the token boundaries on the output side determine exactly what the model is rewarded for getting right in a single forward pass — and therefore define the learning problem itself. The authors use three-digit addition as a controlled testbed to separate input tokenization from output tokenization and show that performance, training dynamics, and internal representations track the output side.

Key Contributions

1. A reframing of tokenization as output supervision. The paper develops the conceptual argument that output token boundaries are the atomic unit of the training loss, so tokenizer granularity defines the granularity of the task a model is trained to solve.

2. A novel decoupling of input and output tokenization. Using a shared vocabulary of 1003 tokens, the authors train a 2x2 factorial set of conditions (FF, FH, HF, HH) so that input and output tokenization can be varied independently with no architectural changes.

3. The minimal computation hypothesis, with probing evidence. The hypothesis states that models resolve what their output supervision demands and are under no direct pressure to resolve more. Linear probes on the residual stream confirm the predicted pattern, and a big-endian variant exposes a gap between what training rewards and what the task requires.

4. A survey of reporting practice. Across 120 recent *CL papers on mathematical or numeric reasoning (ACL, EMNLP, NAACL, EACL main and findings tracks, 2024–2025), 11/120 (9.2%) explicitly mention numeric tokenization strategy, and 83 of the remaining 109 compare models across fragmented and holistic tokenization without flagging it.

Main Findings

• Tokenization affects task performance, replicating prior work. With fragmented tokenization, model M_F converges reliably and fast across all seeds and reaches ceiling. With holistic tokenization, M_H does not converge reliably across seeds and reaches substantially lower accuracy. The paper frames this as a difference in the difficulty of the task each regime defines, not as evidence that one regime is superior.

• The effect is driven by output tokenization, not input tokenization. In the factorial comparison, models sharing an output regime behaved alike regardless of how inputs were encoded: M_F ≃ M_{H_in,F_out} and M_H ≃ M_{F_in,H_out}. Cross-entropy loss was computed only over tokens after the = sign, so the gradient signal was determined entirely by output tokenization.

• Minimal computation holds under little-endian order. Under fragmented output tokenization (M_F, M_{H_in,F_out}), only the first result digit (the units digit, given the little-endian convention) was linearly decodable, with tens and hundreds remaining near chance at every layer. Under holistic output tokenization (M_H, M_{F_in,H_out}), all three result digits became available by the last layer. The pattern tracked the output axis, not the input axis.

• Evaluation accuracy on the held-out 500-sample test set (mean ± standard deviation across the checkpoints probes were trained on): M_F exact match 1.000 (hundreds 1.000, tens 1.000, units 1.000, all ± .000); M_{F_in,H_out} 0.989 (0.997, 0.993, 0.998, ± .012, ± .004, ± .012, ± .002); M_{H_in,F_out} 0.986 (0.999, 0.994, 0.992, ± .009, ± .001, ± .006, ± .007); M_H 0.853 (0.964, 0.922, 0.857, ± .050, ± .012, ± .018, ± .050).

• Random-baseline difficulty is redistributed, not changed in total. Restricted to valid result tokens, a random baseline scores 1/10 per forward pass under fragmented tokenization versus 1/1000 under holistic tokenization, but end-to-end both succeed with probability (1/10)^3 = 1/1000. What differs is that holistic supervision computes one loss over the token 578 with no partial credit, while fragmented supervision decomposes the same target into three 10-way predictions with position-specific credit assignment.

• A tension between gradient reward and task requirement. In big-endian order, carry information propagates right to left, so the currently due digit cannot be computed without resolving later-supervised digits. Even though gradient reward is unchanged, future digits became decodable above chance: final-layer linear probe accuracy for the second result digit rose from 0.11 → 0.29 (M_F) and 0.12 → 0.69 (M_{H_in,F_out}); for the third result digit, from 0.10 → 0.13 and 0.18 → 0.56. Decodability was consistently higher for the adjacent digit than the more distant one, consistent with the "breadcrumbs" hypothesis.

• Reporting practice is a systematic gap. Of the 120 surveyed papers, 20 evaluated only one tokenization type without justifying the restriction, and 6 were unresolvable due to closed-source or undisclosed tokenizers. Even among the 11 papers that mention tokenization, none frames the distinction in terms of output supervision.

Methodology in Plain English

The authors trained small decoder-only transformers from scratch on three-digit addition (e.g., 347 + 231 = 578), holding architecture, data, and optimizer constant and changing only how numbers are tokenized.

The key design choice is a joint vocabulary of 1003 tokens: the integers 0–999, plus + (index 1000), = (index 1001), and PAD (index 1002). Fragmented conditions draw digits only from 0–9; holistic conditions draw whole numbers from 0–999. Because both draw from the same vocabulary, the researchers could tokenize inputs with one function and outputs with another, creating four conditions: fragmented-in/fragmented-out (FF), fragmented-in/holistic-out (FH), holistic-in/fragmented-out (HF), and holistic-in/holistic-out (HH). Loss was masked to output positions only.

Architecture was identical across conditions: 4 layers, d_model = 256, 4 attention heads (d_head = 64), MLP dimension 1024, ReLU activations, no layer normalization, shared embedding and unembedding, roughly 3.4M parameters. Training used teacher forcing and AdamW, for 200,000 steps with batch size 256, across 10 random seeds and a grid of learning rates and weight decays. The best setting shown is lr := 0.0001 and wd := 0.01. A fixed held-out set of 500 problems was excluded from training throughout. Problems were sampled by drawing the result z ~ Uniform(0,999), then x ~ Uniform(0,z), with y = z − x, so that the result space is covered uniformly. Integers were zero-padded to three digits before tokenization.

Two digit orders were used. Little-endian (least-significant digit first) makes addition strictly left-to-right, so gradient reward and task requirement coincide. Big-endian makes the first output digit depend on carries from digits supervised later, separating the two.

To inspect what the models internally represent, the authors trained probing classifiers on residual stream activations at the = token position — the point where the model must have assembled what it needs to start generating. For each condition they used the five checkpoints with highest task accuracy and trained separate probes per digit position (units, tens, hundreds) per layer; linear probes are reported in the main text, with MLP and circular probes in the appendices.

Finally, they surveyed 120 papers from the main and findings tracks of ACL, EMNLP, NAACL, and EACL in 2024 and 2025 whose titles contain "math," "numer," "number," or "arithmetic," manually filtered for topical fit. For each, they searched the full text for the substring "token" and judged whether the digit-level versus multi-digit distinction was explicitly mentioned and whether cross-model comparisons accounted for it.

Why This Matters

Impact on research: The paper claims that when two models are compared on an arithmetic benchmark and one uses holistic output tokenization while the other uses fragmented tokenization, the comparison does not isolate reasoning ability — it conflates reasoning ability with supervision regime. The authors propose that tokenization strategy on task-relevant tokens be reported as a standard model descriptor alongside architecture, parameter count, and training data, that capability comparisons be stratified by output supervision regime where possible, and that mechanistic claims about how models represent structured outputs be qualified as claims about a model under a specific supervision regime.

Real-world applications:

• Benchmark design and model evaluation: Anyone building or interpreting leaderboards for arithmetic, temporal reasoning, code, or morphology needs to know whether models are being compared on the same task or on structurally different ones.

• Tokenizer selection during model development: Teams choosing between digit-wise and whole-number tokenization for a math-heavy application can reason about the trade-off as a supervision decision, not just a compression or vocabulary decision.

• Agentic and tool-use systems that emit structured outputs: Dates, identifiers, literals, and numeric fields all have internal compositional structure, and the granularity at which they are emitted changes what the model must resolve in one pass.

• Interpretability and mechanistic work: Findings about where a model represents an intermediate value are relative to the supervision regime the model was trained under, which matters for anyone publishing circuit-level explanations.

Industry relevance: Production LLMs differ systematically in numeric tokenization granularity — the paper points to Llama 3, Pythia, and OlMo 2 as holistic examples and Qwen 2, Mistral, and Gemma 2 as fragmented examples. Any practitioner comparing such models, or fine-tuning one on numeric tasks, is implicitly choosing a supervision regime. The survey result that 69% of recent numeric-reasoning papers compare across regimes without noting it suggests this is not currently part of standard evaluation hygiene.

Future Directions

1. Direct evidence at production scale. The controlled experiment requires training from scratch because production LLMs share one tokenizer across input and output. The authors argue the core argument is scale-independent (loss is computed over output tokens regardless of model size) and cite indirect evidence from prior work on digit-position-specific circuits in production models, but state that direct evidence at scale remains future work. They note they cannot predict the magnitude of the effect in practice.

2. How output token granularity interacts with post-training and inference regimes. Process supervision, preference optimization over full solutions, RL with outcome rewards, and inference protocols that externalize intermediate results all change what a model is effectively supervised to resolve per forward pass. The paper states this is beyond the scope of its experiment.

3. Whether the gap between gradient reward and task requirement predicts brittleness. The authors ask whether the size of this gap predicts where lookahead-dependent tasks become brittle, naming the systematic lookahead limitation reported for LLMs as one candidate, and leave this as a testable consequence.

4. Extension beyond arithmetic. The authors expect the framework to apply wherever output tokens have internal compositional structure — morphology, date expressions in temporal reasoning, and identifiers and literals in code generation. They note that the morphology picture is unsettled and that the closest existing studies evaluate encoder-only or encoder-decoder models rather than decoder-only ones, so replication of the representational signatures remains an open empirical question.

Target Audience

Researchers and practitioners working on tokenization, arithmetic and numeric reasoning, and mechanistic interpretability of language models will get the most from this paper, as will benchmark designers and anyone who compares model capabilities across models with different tokenizers. The reframing is stated plainly enough to be useful to graduate students and engineers choosing a tokenizer, while the probing methodology and factorial design are aimed at researchers who want to replicate or extend the experiment. Reviewers and meta-analysts of the numeric-reasoning literature will also find the survey of reporting practice directly actionable.

Authors’ abstract

Tokenization in language models is treated by default as an input preprocessing decision. We argue that this framing is incomplete: in autoregressive models, tokenizer granularity determines what the model must resolve in a single forward pass, and therefore the supervision signal it receives. This affects both the difficulty of the learning problem and the representations that emerge inside the model. We test this in a controlled experiment on numeric reasoning with a novel decoupling of input and output tokenization. As the output supervision view predicts, differences in task performance, training dynamics, and model internals are induced by output tokenization and largely invariant to input tokenization. This may matter in practice, because models with different tokenization strategies differ not only in input representation but in the task they were trained on. Comparisons between models may thus partly reflect task definition rather than ability. A survey of 120 recent *CL papers on numeric reasoning confirms that this is rarely acknowledged: only about 10% report the numeric tokenization of the models they evaluate, while 69% compare across tokenization, and thus supervision, regimes without reporting it. While prior work documents that tokenization consistently affects model performance, there is no principled account of why. We argue that framing tokenization as output supervision provides that account.