On the Tip of the Tongue: First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs

AI summary

Overview

Research area: Natural Language Processing — factual recall and hallucination in large language models, with a behavioral probing method inspired by human memory research.

Technical level: Intermediate. The core idea is intuitive, but the paper assumes familiarity with autoregressive decoding, tokenization, entropy-based uncertainty, and standard QA evaluation.

Scope: A behavioral study of whether closed-book factual errors in 16 open instruction-tuned models stem from failures at the first answer token, tested by pre-filling that token and measuring how many errors are recovered across English and multilingual QA benchmarks.

What This Paper Is About

Large language models often answer factual questions incorrectly even when they appear to hold the knowledge, and such errors are usually blamed on missing knowledge in the parameters. This paper asks a narrower question: how many of these errors are failures to start the answer correctly, rather than failures of knowledge itself? The authors borrow the human "tip-of-the-tongue" paradigm — where a speaker who cannot produce a word can still complete it when given its first sound — and test whether giving a model the first token of the gold answer lets it finish an answer it already knows.

Key Contributions

1. Localizing the failure at the first token. The authors show that uncertainty about correctness peaks at the first answer token, and that among gold prefixes of any length, the single first token accounts for the largest share of recovered errors.
2. Quantifying recovery across models and languages. Supplying only the gold first token corrects between a fifth and a half of baseline errors on multi-token answers across 16 open models spanning three families and 270M to 72B parameters, and improves accuracy in all seven languages of a multilingual benchmark.
3. A crossed-cue control separating recognition from cue information. By pairing each question with its own gold first token and with a partner question's incorrect token of the same answer type and matched prior probability, the authors show models continue correct starts more confidently than type-matched incorrect ones.
4. Linking cue recovery to sampled reasoning. Cue-recoverable errors largely overlap with errors the model's own sampled chain-of-thought solves under pass@16, the criterion prior work uses to mark facts as encoded but not recalled.

Main Findings

• Uncertainty concentrates at the first token. For Llama-3.1 8B on TriviaQA, entropy is highest at the first answer token and drops sharply afterward; first-token entropy separates correct from incorrect answers nearly as well as the whole-answer average, while the second and third tokens do much worse.
• First-token entropy predicts errors in all 16 models. AUROC is between about 0.6 and 0.85 on the English benchmarks.
• The cue's success depends on how far the correct token ranked. The cue fixes about two-thirds of errors where the gold first token was the model's second to fifth choice, and under a third of those where it ranked below the top thousand.
• The cue corrects a large share of errors. On questions with at least two gold tokens, the cue corrects between a fifth and a half of baseline errors depending on the model; as a share of baseline errors it corrects between 40 and 50% in every model except the three with the lowest baseline accuracy (Gemma-3 270M, Gemma-3 1B, Qwen-2.5 0.5B).
• Scale reduces error count but not the recovered share. The smallest model of each family gains roughly 15 to 25 points and the largest 10 to 13 points, but the share of errors corrected stays between 40 and 50% even for the 70B and 72B models.
• Only the right cue helps. Random first tokens lower accuracy in every model; a plausible but incorrect first token also lowers accuracy on Llama-3.1 checks; a generic answer prefix leaves accuracy unchanged.
• The first token dominates longer prefixes. For Llama-3.1 8B, the first token alone recovers more errors than all longer gold prefixes combined, and remains the largest group at every answer length in all but the three lowest-accuracy models.
• Gains hold at every answer length. The gain shrinks as answers grow longer but stays positive for answers of five or more tokens.
• Multilingual gains are universal but script-dependent. Cueing improves accuracy for every model in every MKQA language; gains are largest in English and Spanish and smallest in Arabic and Russian. Russian has among the highest baselines of the non-Latin languages yet among the smallest gains, likely because its first token is often a single Cyrillic letter, whereas a single Chinese or Korean character specifies much more of the answer.
• The cue reaches answers outside the model's own candidates. About two-fifths of baseline errors for Llama-3.1 8B and 70B are "IN" (a top-p candidate is a prefix of the gold first token); on those, the cue recovers about half. On "NOT-IN" errors the cue recovers about two-fifths against about a quarter for an oracle over the model's own candidates (Llama-3.1 8B: 40.4 vs. 24.8; 70B: 44.6 vs. 22.6).
• The corrected errors overlap heavily with sampled reasoning. Pass@16 over chain-of-thought samples recovers 77% and 62% of the cue-recoverable errors of Llama-3.1 8B and 70B, against 59% and 47% of all their baseline errors; but only sampled CoT, not direct sampling or greedy CoT, reaches cueing's accuracy.
• Models recognize correct starts they did not produce. In the crossed-cue control with matched priors, every model continues the correct start more confidently in about 60 to 67% of questions (Llama-3.1 8B 66.7, Llama-3.1 70B 65.0, Qwen-2.5 7B 64.2, Gemma-3 12B 60.0), with p < 10⁻¹⁶ for every model. The preference rises with the prior probability of the correct start and falls to about chance when the incorrect start is far more probable.
• Recognition is not the same as discovery. A search over 36 first characters scored by the model (Llama-3.1 8B) gains about as much as beam search, even though two-thirds of questions have a correct candidate: the correct start is continued more confidently than most alternatives but usually less so than the model's own preferred start.
• Judge reliability was checked. The gpt-oss-120b judge agrees with author-checked labels on 93% of 200 sampled MKQA responses (Cohen's κ = 0.86), and errs mostly by rejecting correct responses, including 13 of 14 disagreements.

Methodology in Plain English

The authors introduce First-Token Cueing: they take a question, let the model's chat template format the prompt, then pre-fill the assistant response with the first token of the gold answer and let the model continue greedily. They compare this against a plain baseline using the same prompt and decoding. Because the cue comes from the gold answer, this is a diagnostic probe, not a deployable method — it measures which errors are recoverable, not how to fix them at inference time.

Data and models. They evaluate three English open-domain QA benchmarks (TriviaQA, NQ-Open, WebQuestions) and MKQA in seven languages (English, Spanish, Russian, Arabic, Japanese, Chinese, Korean), using 16 instruction-tuned models from Gemma 3, Llama 3.1/3.2, and Qwen 2.5, served with vLLM on a single NVIDIA B200 GPU with a 4,096-token context window in bfloat16.

Headline subset. When the gold answer is a single token, the cue is the answer, so the main results restrict to questions whose gold answer has at least two tokens — excluding about a tenth of the English questions and fewer of the MKQA questions. Here the model must generate the rest of the answer itself.

Grading. A judge model (gpt-oss-120b) grades responses against the gold answer and its aliases; alias-aware exact match and token-level F1 reproduce the gains as a check.

Controls. They replace the gold token with random tokens, plausible-but-wrong tokens, and generic answer prefixes. They also build a first-token oracle that prefills every top-p (p = 0.95) candidate and completes it, to see how much recovery is available from the model's own candidate set.

Crossed-cue design. To separate "the model recognizes the answer" from "the cue leaked information," they pair each question with another question from the same dataset and the same answer type (labeled by gpt-oss-120b into seven types such as person, location, date), matching the two first-token probabilities as closely as possible. Each token serves as the correct cue for one question and the incorrect cue for the other, canceling token-specific effects. They then compare continuation confidence (negative mean entropy) after each start.

Why This Matters

The paper reframes a chunk of factual hallucination as a decoding-step problem rather than a knowledge-storage problem, and it points to a specific place — the first answer token — where inference-time compute might be best spent. It also supplies a lightweight behavioral test that requires only prompt pre-filling, no access to internal states.

Real-world applications:

• Retrieval-augmented QA: retrieval could supply candidate answer starts rather than whole answers, with the model completing and verifying them.
• Confidence flagging in assistants: first-token entropy is a cheap, single-position signal for routing high-uncertainty questions to search, human review, or clarification.
• Multilingual deployment: the per-language and per-script results tell practitioners where a one-token intervention carries little information (e.g., Cyrillic scripts) and more search is needed.
• Test-time compute allocation: concentrating sampling or branching at the first decoding step instead of treating every step alike.

Industry relevance: the finding that scale reduces error counts but not the share of recoverable errors is directly relevant to teams deciding whether to scale models or add inference-time mechanisms, and the fact that models recognize correct starts they cannot find on their own sets a realistic ceiling for reranking-only approaches.

Future Directions

• Supplying the first token without the gold answer. The authors explicitly name retrieval as a way to propose starts outside the model's own candidate set, and note that whether such methods recover First-Token Cueing's gains remains open.
• Extending the probe beyond short-answer QA. The counterpart in long-form generation would be the first token of each factual claim, which the paper lists as a natural next step.
• Choosing among starts. Because confidence favors the correct start over an incorrect one of similar prior probability but not over the model's own preferred start, selection will likely need evidence from outside the model.
• Moving from behavioral to internal evidence. The authors identify recall failures behaviorally, through cue recovery and its overlap with reasoning-based recovery, rather than from internal states — leaving a mechanistic account unaddressed.

Target Audience

Researchers and practitioners working on factual hallucination, knowledge editing, decoding-time interventions, and uncertainty quantification in LLMs; multilingual evaluation teams; and cognitive scientists interested in behavioral analogies between human tip-of-the-tongue states and model recall failures. Readers need comfort with tokenization, greedy decoding, and AUROC-style evaluation, but the experimental logic is accessible without a mechanistic-interpretability background.

Authors’ abstract

Large language models often answer factual questions incorrectly in closed-book settings, and such errors are usually attributed to missing knowledge. Yet models often encode answers they fail to produce. We show that many closed-book errors are first-token recall failures: the model commits to a wrong first token, but once given the right one it completes the answer and often recognizes it as correct. We study this with First-Token Cueing, a behavioral probe that prefills the response with the first token of the gold answer. Model uncertainty peaks at the first answer token, where it already predicts correctness, and among gold prefixes of any length the first token alone accounts for the largest share of recovered errors. Across 16 open models, this one token corrects between a fifth and a half of the errors on multi-token answers in English QA and improves accuracy in all seven languages of a multilingual benchmark. Most of the errors it corrects are also solved by the model's own sampled reasoning, a criterion for encoded knowledge, and models recognize correct starts they do not produce: they continue a correct start more confidently than an incorrect start of the same answer type and prior probability, yet rarely find it without the gold answer. These results locate a recoverable class of factual errors at the first decoding step, where inference-time methods may gain the most.