Feeling of Knowing in Large Language Models

AI summary

Overview

Research area: Large language model reliability and metacognition — specifically, whether an LLM's judgment about what it can answer stays aligned with its actual knowledge after post-training updates.

Technical level: Intermediate. The paper assumes familiarity with supervised fine-tuning, LoRA adapters, preference/policy optimization, and AUROC-based evaluation, but its central idea (a model knowing what it knows) is explained in accessible terms.

Scope: A study of "functional feeling of knowing" (FoK) in LLMs, its layer-wise localization, its breakdown under knowledge updates, and a three-stage training framework for realigning it — evaluated on factual QA with Qwen2.5 and LLaMA-3.1 backbones.

What This Paper Is About

When an LLM is fine-tuned or edited to learn new facts, what it can answer changes — but its internal judgment about whether it can answer may not change with it. The paper calls this mismatch knowledge–judgment desynchronization and measures it with a scalar "functional feeling of knowing" score: the normalized probability the model assigns to "yes" when asked whether it can answer a question correctly. The goal is to train a separate, dedicated judgment channel that tracks the model's current knowledge state, and then to use that judgment to decide whether to answer or to explain why it cannot.

Key Contributions

1. Diagnosis of functional FoK after knowledge updates. Using ROME-style causal tracing across Qwen2.5 sizes from 0.5B to 32B, the authors locate the FoK signal in late layers (about two-thirds of model depth), later than the middle-layer factual associations ROME reports (about one-third of depth). They then show that knowledge SFT alone raises accuracy but degrades FoK AUROC in six of seven tested models.

2. A two-channel framework with three training stages. The knowledge channel is a parameter delta (κ) trained by On-policy Supervised Fine-Tuning (OP-SFT); the FoK channel is a separate delta (φ) trained by Knowledge-delta FoK Training (KΔ-FT) across the OP-SFT checkpoint trajectory; and FoK-guided Policy Optimization (FGPO) trains the model to reason, commit to a yes/no judgment, and then answer or explain.

3. Evidence that FoK is context-dependent. Re-querying Qwen2.5-7B at increasing prefixes of its own MATH solution traces shows FoK scores separating toward 0 and 1 and prefix-level AUROC rising — motivating a generate-context-then-judge procedure.

4. Evaluation beyond a single update. Cross-dataset tests (SelfAware, Natural Questions, WebQuestions), three sequential TriviaQA updates, seed variation (test AUROC 0.777 ± 0.009 over four seeds for Stages 1–2; 0.811 ± 0.003 over three FGPO seeds from one shared initialization), and separate MMLU / ARC-Easy capability checks.

Main Findings

• Knowledge training breaks the judgment, and not in one direction. In Table 1, K-SFT raises accuracy for every model, but AUROC drops for six of seven — the largest drop on Qwen2.5-3B (−0.220). Some models shift toward omission (refusing questions they would answer correctly); larger Qwen models better preserve AUROC but shift toward illusion (claiming they can answer when wrong).

• Training judgment alone is cheap. Judgment SFT (J-SFT) improves AUROC substantially (e.g., +0.255 on Qwen2.5-1.5B, +0.213 on Qwen2.5-0.5B) with small accuracy changes, and applying J-SFT after K-SFT recovers FoK while preserving most of K-SFT's accuracy gain.

• Naive knowledge updates damage held-out accuracy and every baseline estimator. After vanilla K-SFT on TriviaQA with Qwen2.5-7B, in-domain accuracy rises from 0.680 to 0.874 but held-out accuracy falls from 0.672 to 0.546, and held-out AUROC falls from 0.688 to 0.519. Confidence-based estimators become illusion-dominated; consistency-based estimators trade illusion for large omission.

• The two-channel method recovers FoK on the updated model. The full method reaches held-out accuracy 0.692, AUROC 0.809, and Bias 0.308 — held-out AUROC matching Base + EliCal's 0.810 on the un-updated model, while held-out accuracy exceeds the 0.672 Base value. Update-set accuracy is 0.961 with AUROC 0.817.

• OP-SFT alone is gentler on FoK than vanilla K-SFT. Held-out AUROC after OP-SFT alone is 0.603 versus 0.519 after vanilla K-SFT, attributed to supervising only the corrected answer span on the model's own sampled responses.

• FoK adaptation helps even when accuracy is already preserved. Starting from the same OP-SFT checkpoint, direct judgment reaches 0.603 AUROC, Prob 0.571, EliCal 0.618, and KΔ-FT 0.770.

• A frozen FoK channel decays across sequential updates; a retrained one does not. Over three rounds, the frozen φ₁ falls from 0.798 to 0.754 AUROC, while per-round retrained φᵢ stays near 0.80 (0.798, 0.800, 0.797). Held-out accuracy stays below the 0.672 Base value across all three rounds, so cumulative accuracy improvement is not demonstrated.

• KΔ-FT is the main source of the FoK gain. In the held-out ablation, AUROC goes 0.688 (Base) → 0.603 (+OP-SFT) → 0.770 (+KΔ-FT) → 0.809 (+FGPO), with illusion falling from 0.683 to 0.432 and Bias from 0.402 to 0.308.

• Threshold behavior splits by stage. On Qwen3-8B in thinking mode, Base and Vanilla K-SFT produce nearly flat coverage–accuracy curves; OP-SFT + KΔ-FT restores a monotonic trade-off; FGPO flattens them again because its judgments collapse to near 0 or 1, making the threshold irrelevant once the policy commits.

• Capability is roughly preserved. MMLU is 0.711 and ARC-Easy 0.830 for the full method, versus 0.687 / 0.811 for Base and 0.713 / 0.840 for vanilla K-SFT. Notably, Base + EliCal drops ARC-Easy to 0.504 while raising MMLU to 0.709.

Methodology in Plain English

The authors treat "can I answer this?" as its own trainable skill, separate from knowing facts, and give it its own small set of adapter weights.

First, they update knowledge carefully. Instead of training on fixed question–answer pairs, they let the model generate 32 responses per question, keep the model's own wording, and fix only the wrong answer span — applying loss only to that corrected span. This preserves the model's surrounding style and, empirically, its existing judgment ability. Checkpoints are saved along the way, producing a trajectory of intermediate knowledge states.

Second, they train the judgment channel across that whole trajectory rather than at a single endpoint. At each saved checkpoint, the label is simply whether that checkpoint actually answers the question correctly, so the judgment learns to follow answerability as it shifts from pre-update to post-update, instead of memorizing a static label.

Third, they train the model to act on its judgment. The model produces a short reasoning trace, then a yes/no verdict, then either an answer or an explanation of what it lacks. A composite reward scores the verdict against ground-truth answerability, scores the answer or explanation conditioned on that verdict, adds format and termination terms, and subtracts a length penalty. Wrong answers and malformed outputs are penalized more heavily than valid abstentions are rewarded, so the model is not pushed to game the system by refusing.

Evaluation uses AUROC over the normalized "yes" probability, plus two thresholded error rates at J > 0.5: illusion (says yes but is wrong) and omission (says no but would be right), combined into a signed Bias score in [−1, 1]. Answer correctness on TriviaQA is checked by Qwen2.5-32B-Instruct as an LLM judge.

Why This Matters

Impact on research. The paper reframes answerability judgment as something that must be co-trained with each knowledge update rather than calibrated once against a frozen checkpoint. That connects two literatures that usually run in parallel — uncertainty/abstention methods and knowledge editing/continual learning — and provides a diagnostic (AUROC before and after an update) that any editing method can be tested against. The layer-depth finding also gives a concrete, testable dissociation between where facts live and where judgments about facts live.

Real-world applications:

• Retrieval triggering: only call a retrieval system when the model's FoK judgment says its current knowledge is insufficient, which is exactly what breaks after domain fine-tuning.
• Model routing: escalate to a larger or stronger model when the updated model judges it cannot answer, rather than routing on the base model's old confidence profile.
• Calibrated abstention in deployed assistants: avoid confidently wrong answers after a model is fine-tuned on new product, policy, or clinical content.
• Continual-learning pipelines: monitor whether each update round has desynchronized judgment, and retrain the judgment channel per round rather than freezing it.

Industry relevance. Fine-tuning and knowledge editing are standard deployment practice, and the paper shows the two standard side effects — degraded held-out accuracy under vanilla K-SFT (0.672 → 0.546) and collapsed FoK estimators — appear together. A lightweight adapter-based fix that costs a modest MMLU/ARC-Easy trade (0.687/0.811 → 0.711/0.830) is directly actionable for teams shipping updated models.

Future Directions

• Longer update sequences and other task types. The paper explicitly limits itself to short-answer factual QA with gradient-based updates and three sequential rounds; procedural tasks such as code generation and mathematical proof, and longer update chains, are untested.
• Non-gradient editors. The MEMIT test in the appendix checks that desynchronization occurs under one such editor, but not whether the full framework repairs it. Extending the two-channel approach to edit-based updates is open.
• Backbones with weak baseline FoK. Models whose baseline FoK signal is near chance are outside the tested scope, leaving open whether the framework can create judgment ability rather than realign it.
• Causal status of the context effect. The cue analyses lack a matched pretrained-model control, so it is unresolved whether training induces sensitivity to accessible context or merely preserves sensitivity already present in the backbone.

Target Audience

Researchers and engineers working on LLM uncertainty, abstention, hallucination mitigation, knowledge editing, and continual learning — particularly those who fine-tune or edit models in production and need answerability judgments to survive those updates. It is also useful for readers interested in the layer-wise organization of LLM capabilities, and for anyone evaluating whether metacognitive claims about LLMs are supported by behavioral evidence rather than assumed.

Authors’ abstract

Post-training updates change what large language models (LLMs) can answer, but may leave their answerability judgments misaligned. We study this knowledge-judgment desynchronization through functional feeling of knowing (FoK), which supports selective generation, retrieval triggering, and model routing without implying subjective experience. We propose two trainable channels: a knowledge channel that changes answerability and a FoK channel that judges the current knowledge state. Three stages train them: on-policy supervised fine-tuning (OP-SFT) updates knowledge from corrected on-policy answers; knowledge-delta FoK training (K$Δ$-FT) adapts judgments across intermediate knowledge states; and FoK-guided policy optimization (FGPO) trains the subsequent answer or explanation. Factual-QA experiments, cross-dataset evaluation, and three sequential updates show that explicit FoK alignment improves judgment quality on updated knowledge. MMLU and ARC-Easy separately assess capability preservation.