What's the plan? Metrics for implicit planning in LLMs and their application to rhyme generation and question answering

AI summary

Overview

Research area: Mechanistic interpretability and language model controllability — specifically whether autoregressive LLMs "plan ahead" internally for tokens they have not yet generated, studied through rhyme generation and question answering.

Technical level: Advanced. The paper assumes familiarity with residual stream activations, activation steering, attention heads, MLPs, activation patching, and logit differences, although the proposed method is deliberately simpler than prior cross-layer transcoder work.

Scope: A 1-sentence scope: The paper introduces quantitative metrics and a simple mean-activation-difference steering method to detect and manipulate implicit forward and backward planning in 23 open-weight LLMs (1B–32B parameters), applying them to rhyming couplet generation and noun question answering.

What This Paper Is About

Prior work (Lindsey et al., 2025) argued qualitatively that Claude 3.5 Haiku implicitly plans the rhyme of the next line before generating it, using a cross-layer transcoder — an approach that is expensive, hard to replicate, and limited to one closed-source model. This paper asks whether such planning can be detected and measured much more simply, across many models, and whether it extends beyond poetry to question answering. The goal is to build datasets, metrics, and a lightweight steering intervention that make implicit planning a measurable property of a model rather than a collection of persuasive examples.

Key Contributions

1. Two new datasets. A rhyming dataset of 1050 first lines spanning 10 rhyme families and 20 rhyme-family pairs, and a question answering dataset of 500 questions targeting 20 intended answer nouns arranged into 20 noun pairs (one vowel-initial, one consonant-initial, e.g. eye / heart).

2. Quantitative metrics for planning. Metrics for successful forward planning (e.g. Fraction of Correct Rhyme Family, Fraction of Correct Answer, and their steered variants), backward planning (Fraction of a/an), and successful backward planning (Fraction of Correct Last Word Regeneration and its steered variant), plus probability-based metrics.

3. Evidence across 23 open-weight models, 1B to 32B parameters. Four model families (Gemma2, Gemma3, Qwen3, Llama3.1/3.2), with both base and instruction-tuned variants; the paper reports planning as a universal mechanism present in smaller models than previously thought, starting from 1B parameters.

4. Circuit-level observations. An analysis of the rhyme planning circuit in Gemma2 9B, isolating two attention heads (L30H3 and L31H15) and downstream MLP layers 30–39, plus the finding that question answering appears to use a similarly shaped but distinct circuit.

Main Findings

• Larger and instruction-tuned models rhyme better. Baseline rhyming ability generally increases with model size, and instruction-tuned models rhyme more consistently than their base versions. The authors state this is not attributable to tokenization differences between models (Appendix G).

• Backward planning is detectable in all models via regeneration. When the last word of the second line is regenerated with the first line removed, lines originally generated in a rhyming context are completed with a word from the intended rhyme family above chance, for all models studied. The chance baseline is estimated as the average frequency of the other rhyme families in that model's stochastic sampling.

• Steering on a single token changes the rhyme produced later. Applying a mean activation difference vector at the last word of the first line shifts the rhyme family of the second line to rates comparable to the model's own baseline rhyming rate. The exception: for models with the lowest rhyming ability (base variants of Gemma3 1B and Llama 3.2 3B), steered rhyming is substantially lower than unsteered.

• Steered regeneration matches baseline regeneration. Regeneration rates toward the target rhyme family under steering are close to baseline regeneration rates, which the authors take as evidence that steering does not merely swap the final word but re-shapes the intermediate words that lead to it.

• Two steering positions behave qualitatively similarly. All models can be steered on the last word of the first line, typically in lower layers. Some models can also be steered on the newline token, but this effect is only pronounced for Gemma2 9B and Gemma3 27B (both instruction-tuned and base variants). Newline steering is somewhat less effective than last-word steering yet yields comparable planning metrics. The proposed explanation is that some information bypasses the steered newline token and flows directly from the unsteered last word position onward.

• Question answering shows forward and backward planning across all 23 models. Steering toward one noun of a pair shifts answer frequencies, and it also shifts article choice before the noun is generated: models decrease "an" and increase "a" when steered toward a consonant-initial noun in a question about a vowel-initial noun. On neutral questions, instruction-tuned models showed a stronger steering effect.

• A more complex QA task also shows planning. In a small-scale experiment, Gemma3 27B — the strongest model in the study — shows signs of implicit planning multiple tokens ahead, with backward planning evidenced by subject-verb agreement (Appendix L).

• A rhyme planning circuit can be localized. In Gemma2 9B (instruction-tuned), steering was applied on the newline token at layer 27. Activation patching of attention heads L30H3 and L31H15 onto the last token recovers most of the steering effect (59% to 93% across the four analyzed couplets). These heads attend to the last word of the first rhyming line and the following newline token but not other tokens; the copied information is then converted into predictions in subsequent MLP layers 30–39, and patching those MLPs recovers the steering effect almost entirely.

• Rhyme and QA use different circuits. The attention heads that transfer rhyming information contribute little in question answering, where most of the action happens in later layers; attention head L39H13 appears particularly important there, with final MLP layers driving the output distribution.

• All rhyming metrics correlate. The paper reports that rhyming ability goes hand in hand with planning for a rhyme across logically independent aspects: how early the planning circuit activates, how much influence it has on output logits, and how well it boosts tokens leading to the correct rhyme family (Appendix E).

• Training data efficiency varies by capability. Additional training examples mostly benefit less capable models, while the most capable models generalize from a few examples or even a single example pair (Appendix B, Figure 7). For fair comparison across models, 85 training examples per rhyme family were used.

Methodology in Plain English

The authors look at two tasks where a model must commit to something now that only pays off several tokens later.

In the rhyming task, a model is given the first line of a couplet and must produce a second line that ends in a rhyme. The authors assume the rhyme must be decided at the end of the first line — before any words of the second line exist. They build a steering vector by comparing average hidden activations at a chosen position (the last word, or the newline token after it) between lines from two different rhyme families, then multiply the difference by a constant m = 1.5. Adding this vector to a single token position during generation nudges the model toward the second rhyme family. They pick, for each model and rhyme-family pair, the layer and position where steering works best, then measure how often the target rhyme family actually appears.

The question answering task is designed so planning is visible in a second way: each noun in a pair forces a different indefinite article before it (an eye vs. a heart). If the model plans the noun before writing it, steering toward the consonant-initial noun should make it use "a" earlier in the answer.

To test backward planning — whether the plan actually shapes the intermediate text — the authors strip the first line away and regenerate just the last word of the second line. If the model had planned the rhyme, the regenerated word should still tend to come from the intended family. Finally, for the Gemma2 9B circuit analysis, they replace the activations of specific attention heads with the activations those heads would have had under steering, and check how much of the steering effect is reproduced on the output logits.

Why This Matters

Implicit planning is a safety-relevant property: if a model can commit internally to an output long before producing it, then intermediate text is not a faithful window into what the model is doing. The paper's main methodological message is that detecting this does not require expensive dictionary learning — mean activation difference steering plus a small labelled dataset is enough, which makes the analysis reproducible and scalable to models that researchers can actually download.

Real-world applications implied by the work (the paper does not demonstrate deployed systems):

• Controllable creative writing: steering rhyme and rhythm without hard phonetic constraints, letting the model generate naturally while shifting the target rhyme family.
• Auditing and evaluation: using steered metrics as a comparable capability probe for planning across model families and checkpoints, including small models.
• Detecting hidden commitments in safety-critical deployments: noticing when a model has already settled on an answer or behavior before the visible output reveals it.
• Interpretability tooling: reusing the single-token steering and patching recipe to localize task-specific circuits beyond rhyme and QA.

Industry relevance: the method is cheap relative to cross-layer transcoder training, which the paper notes requires days of compute on an H100 for even a modest-size model. That cost profile makes planning measurement feasible as a routine evaluation, and the paper's reproducibility materials (data, code, and outputs on OpenReview and a public GitHub repository) make third-party replication straightforward.

Future Directions

• Why do some models use the newline token? Gemma2 9B, Gemma3 27B, and Claude Haiku appear to have a rhyming circuit that involves the newline position. Whether this more elaborate circuit is what drives their quantitatively better planning behavior remains an open question.

• Do other models have a few dominant rhyme-planning heads? The finding that a small set of attention heads implements rhyme planning has so far been established only for Gemma2 9B.

• Transfer to other long-distance dependencies. The authors propose applying the metrics to instruction following and chain of thought, and to other intervention methods besides mean activation difference steering.

• More complex planning tasks. The subject-verb agreement experiment with Gemma3 27B is described as small scale; extending planning analysis to tasks with longer, more structured dependencies is left open.

The provided paper text states that section 5 discusses limitations and further directions, but the truncated content does not include the specifics of that limitations discussion.

Target Audience

Researchers in mechanistic interpretability and LLM controllability who want a cheap, reproducible alternative to transcoder-based planning analysis; NLP researchers studying text generation with global constraints such as rhyme and metre; evaluation and safety teams who need a scalable probe for whether a model commits to outputs before emitting them; and graduate students looking for a worked example of designing quantitative metrics around an interpretability hypothesis.

Authors’ abstract

Prior work suggests that language models, while trained on next token prediction, show implicit planning behavior: they may select the next token in preparation to a predicted future token, such as a likely rhyming word, as supported by a prior qualitative study of Claude 3.5 Haiku using a cross-layer transcoder. We propose much simpler techniques for assessing implicit planning in language models. With case studies on rhyme poetry generation and question answering, we demonstrate that our methodology easily scales to many models. Across models, we find that the generated rhyme (e.g. "-ight") or answer to a question ("whale") can be manipulated by steering at the end of the preceding line with a vector, affecting the generation of intermediate tokens leading up to the rhyme or answer word. We show that implicit planning is a universal mechanism, present in smaller models than previously thought, starting from 1B parameters. Our methodology offers a widely applicable direct way to study implicit planning abilities of LLMs. More broadly, understanding planning abilities of language models can inform decisions in AI safety and control.