Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuning

AI summary

Overview

• Research area: Natural Language Processing — supervised fine-tuning (SFT) of large language models, factuality and hallucination mitigation.
• Technical level: Intermediate. The paper assumes familiarity with SFT, instruction tuning, and long-form factuality evaluation (claim decomposition, evidence retrieval, FActScore), but its central idea — matching training targets to what the base model already knows — is explained from first principles with formal definitions and diagrams.
• Scope in one sentence: The paper reframes hallucinations introduced during SFT as a mismatch between the factual claims required by training targets and the base model's parametric knowledge, then compares existing and newly proposed methods for constraining SFT targets to that knowledge boundary using Qwen 3 4B and OLMo 3 7B.

What This Paper Is About

Supervised fine-tuning teaches a base model to imitate target responses, and those targets often contain factual claims the model never robustly internalized during pre-training. The authors argue this mismatch pushes the model to produce plausible-sounding but unsupported claims, and that it also teaches a general response policy of answering confidently under uncertainty rather than respecting the model's knowledge boundary. The goal is to test whether constraining SFT targets to only what the base model already knows — "knowledge-aligned SFT" — reduces measured hallucinations without breaking the model's general abilities.

Key Contributions

1. A formal framework for knowledge-aligned SFT. The authors define the dataset knowledge 𝒦(𝒟) as the union of atomic claims extracted from prompt–response pairs, define a claim as "known" if it lies in the base model's parametric knowledge 𝒦(M_base), define the "hallucination zone" as generations falling outside real-world knowledge 𝒲, and define knowledge-aligned SFT as any procedure that constructs a dataset 𝒟* whose responses contain only known claims.
2. Two new knowledge-alignment methods. Evidence Rewrite extends generation-based alignment (FLAME) by verifying base-model generations against retrieved external evidence before rewriting. Recall Rewrite, the primary contribution, probes whether the base model can consistently recall each knowledge-dependent claim via generated probing questions, answer sampling, and entailment checking, keeping only claims classified as consistently recalled.
3. A controlled, unified comparison. FLAME (generation-based) and UNIT_cut (estimation-based) are compared against both new methods under identical training settings, alongside standard SFT baselines including a Tülu 3 large-data SFT baseline.
4. A causal ablation on the share of known claims. Section 4.4 varies only the proportion of knowledge-dependent claims classified as known (%Known) at 100%, 50%, and 0%, holding prompts, the number of non-refusal (1,777) and refusal (302) training examples, and all other training settings fixed. The trained Recall Rewrite data and all intermediate pipeline outputs are released for both base models.

Main Findings

• Knowledge-aligned SFT reduces measured hallucinations. On Qwen 3 4B, Recall Rewrite reaches 84.2 %Supp. and 84.1 FActScore on WildHalu and 56.2 %Supp. and 76.4 FActScore on Bios, versus 76.6 %Supp. / 74.4 FActScore and 36.0 %Supp. / 34.1 FActScore for standard SFT on OASST1. All knowledge-aligned variants except FLAME consistently outperform standard SFT on %Supp. and FActScore.
• Naive self-generation is not enough. FLAME (6, 7,882, 73.0, 74.4 on WildHalu) does not improve over standard SFT on OASST1 (2, 8,059, 76.6, 74.4), suggesting generated content is an unreliable proxy for parametric knowledge. Adding verification (Evidence Rewrite: 1, 7,842, 80.1, 78.3) improves over FLAME.
• Recall Rewrite gives the largest factuality gains but the lowest coverage. It produces fewer supported claims (#Supp. 6,157 on WildHalu and 2,561 on Bios) and substantially more refusals (#Ref. 55 and 252) than other OASST1-trained variants. The authors state its higher FActScore should not be read as improved factual generation at equal coverage, but as a more conservative response policy.
• There is a coverage–factuality trade-off. Stricter filtering yields more reliable but less informative outputs. The %Known ablation confirms this directly: at 100% known claims the model achieves 84.7 %Supp. and 86.1 FActScore on WildHalu with 97 refusals, at 50% it drops to 78.7 and 80.4 with 45 refusals, and at 0% to 79.0 and 79.5 with 22 refusals.
• Recall Rewrite refuses more reliably on unanswerable questions. On UnknownBench it achieves the highest recall on all three subtasks (64.1 on FalseQA, 66.3 on NEC, 79.1 on RefuNQ) and the best F1 on all three (68.7, 68.8, 69.9), while exhibiting the lowest precision (74.0, 71.4, 62.6).
• General capabilities are largely preserved. Recall Rewrite's OLMES average is 68.9 versus 69.8 for standard SFT on OASST1 — a 0.9-point difference within the 2.1-point band spanned by all OASST1 models. Its largest per-benchmark gaps (GSM8K −2.3, IFEval −3.2) are of the same magnitude as differences among the baselines themselves (FLAME −3.2 on IFEval, Evidence Rewrite without brainstorming −5.4 on GSM8K).
• The IFEval gap is explained by refusals. Recall Rewrite refuses 30 of the 541 IFEval prompts, mostly creative-writing tasks; on the remaining prompts the two models are on par (56.3 vs. 56.5). The GSM8K errors are described as ordinary reasoning slips without refusals.
• Knowledge-aligned SFT is competitive with additional post-training stages. On OLMo 3 7B, Recall Rewrite trained with SFT alone on OASST1 reaches 82.7 %Supp. and 82.5 FActScore on WildHalu, higher than the official SFT (77.9 / 75.6), DPO (76.6 / 76.5), and RLVR (78.5 / 78.4) checkpoints trained on Dolci, while keeping a comparable number of supported claims.
• The official Qwen 3 Instruct model remains stronger. It reaches 85.9 %Supp. / 87.1 FActScore on WildHalu and 60.5 / 79.3 on Bios, but the comparison is uncontrolled (differences in instruction data scale and composition, post-training stages, and possible benchmark contamination). Differences between Recall Rewrite and Qwen 3 Instruct were not statistically significant (p > 0.05) on either dataset.
• Data scale alone does not fix the problem. The Tülu 3 SFT baseline, trained on a much larger data mixture, scores 79.1 %Supp. / 77.2 FActScore on WildHalu and 34.4 / 33.3 on Bios.
• Claim retention statistics. Under the default filter (j_e/k_e/j_c/k_c = 2/1/2/1), 79.4% of knowledge-dependent claims are classified as known for Qwen3-4B-Base, so that together with non-knowledge-dependent claims 87–88% of all claims survive the rewrite.

Methodology in Plain English

The researchers start from a base model and an SFT dataset of prompt–response pairs. They break each response into atomic claims, keeping only claims that are factual, procedural, or structural and that are not already given in the prompt — that is, claims the model must supply from its own knowledge. This includes implicit meta-knowledge, such as the three-line structure and syllabic constraints needed to write a haiku.

Since a model's knowledge cannot be observed directly, each method approximates it differently:

• FLAME (existing, generation-based): replace gold responses with the base model's own generations, assuming self-generated content is constrained to what the model knows. Non-knowledge-seeking prompts (e.g., summarization) keep the gold response.
• UNIT_cut (existing, estimation-based): keep only claims whose claim-conditioned probability (a token-level confidence signal) exceeds a threshold.
• Evidence Rewrite (new): sample a long-form base-model response, decompose it into claims, retrieve Wikipedia evidence, verify each claim as supported or unsupported, and have a rewriter model compose a fluent response from only the supported claims. If those claims are insufficient to address the prompt, the rewriter is instructed to refuse. A brainstorming step is added before fact-checking to counter response shortening.
• Recall Rewrite (new): take each knowledge-dependent claim from the gold response, have a teacher model generate five diverse, context-independent probing questions (J=5), sample two answers per question from the base model (K=2) at temperature 0.5, and check whether each answer entails, contradicts, or is unrelated to the claim. A question is "entailing" if at least k_e answers entail and "contradicting" if at least k_c answers contradict. A claim counts as consistently recalled if at least j_e questions are entailing AND at most j_c questions are contradicting. The default setting is 2/1/2/1.

All pipeline steps for Evidence Rewrite use gpt-4o-mini; Recall Rewrite uses gpt-5-mini for claim decomposition, question generation, entailment checking, and response rewriting, with only the answers sampled from the base model. Training uses the English subset of Open Assistant 1 (OASST1), specifically the first turn of each conversation tree, giving 3,468 data points, trained via the TRL library with consistent hyperparameters across variants. Evaluation follows the framework of Wu et al. (2025) on WildHalu (500 real-world entities) and Biography (500 people with Wikipedia pages), using gpt-4 for claim decomposition and gpt-4o-mini for claim verification, reporting #Ref., #Supp., %Supp., and FActScore.

Why This Matters

Impact on research. The paper unifies generation-based and estimation-based hallucination mitigations under a single formalism, which makes them directly comparable rather than studied in isolation. Its controlled %Known ablation reproduces the causal effect that fine-tuning on facts unknown to the base model drives hallucination, but does so at the claim level on real open-domain instruction data rather than in closed-book QA. It also reports a concurrent finding (Calderon et al., 2026) that recall, rather than encoding, is the main bottleneck for parametric factuality.

Real-world applications:

• Instruction-tuned assistants that answer detailed questions in domains where users cannot easily detect errors — the paper's motivating failure case.
• Long-form entity description, such as biography generation, where long-tail facts are common and refusal rates spike (Recall Rewrite produces 252 refusals on the Bios benchmark versus 4 for standard SFT on OASST1).
• Systems that must recognize unanswerable questions — the UnknownBench results show improved refusal recall on false premises (FalseQA), fabricated concepts (NEC), and questions with replaced nouns (RefuNQ).
• Data curation pipelines for teams that assemble their own instruction-tuning mixtures and want to control the factual content of supervision.

Industry relevance. The paper notes that Recall Rewrite's pipeline has API costs (reported in Table 10) and that applying this control to large instruction-tuning pipelines "will require cheaper approximations of the knowledge probing." Knowledge-aligned rewriting makes the factual content of SFT targets a controllable variable, which is directly relevant to anyone building post-training pipelines where hallucination is a product-risk rather than an academic metric. The finding that knowledge-aligned SFT matches or exceeds factuality-oriented DPO and RLVR checkpoints on one benchmark (WildHalu) with a single SFT stage is relevant to teams that cannot afford multi-stage post-training.

Future Directions

1. Cheaper approximations of knowledge probing. The authors state explicitly that scaling this control to large instruction-tuning pipelines requires cheaper alternatives to the current question-generation and entailment pipeline.
2. Interactions with later post-training stages. The paper focuses on SFT and does not fully characterize how knowledge-aligned SFT interacts with DPO or RLVR. The OLMo comparison suggests factuality-oriented stages help, but whether the gains stack remains open.
3. Beyond binary known/unknown. The limitations section notes that the framework treats factual knowledge as binary, whereas real models may exhibit graded confidence, partial knowledge, or sensitivity to phrasing; modeling these distinctions explicitly could improve data construction.
4. Broader evaluation beyond entity-centric benchmarks. WildHalu and Biography both elicit descriptions of a single named entity. The authors note that neither tests prompts whose factual content is not organized around one entity, and that the automatic fact-checking stages can introduce errors on underspecified or domain-expertise claims.
5. Testing the ablation across more models and datasets. The %Known ablation currently manipulates only one base model and one dataset while holding training set size and refusal ratio fixed.

Target Audience

Researchers and practitioners working on supervised fine-tuning, instruction tuning, and hallucination mitigation will get the most from this paper, particularly those who build SFT data mixtures and want a principled way to decide which factual targets to keep. It is also relevant to evaluation researchers interested in long-form factuality benchmarks and refusal behavior, and to engineers comparing the cost-effectiveness of data-side interventions against additional post-training stages such as DPO and RLVR. Readers should be comfortable with terms like parametric knowledge, atomic claim decomposition, and FActScore, though the formal framework in Section 2 defines its notation explicitly.

Authors’ abstract

Supervised fine-tuning (SFT) trains a base language model to imitate target responses, and these targets may require knowledge the base model has not robustly internalized. We study this as a source of hallucinations and frame a group of mitigation methods as \emph{knowledge-aligned SFT}: constraining SFT training targets to the base model's parametric knowledge. Under a unified setup, we compare existing generation-based and estimation-based knowledge-alignment methods and introduce two new variants: Evidence Rewrite, which verifies base-model generations using external evidence, and Recall Rewrite, which retains claims only when they can be consistently recalled by the base model. Experiments with Qwen 3 4B and OLMo 3 7B show that knowledge-aligned SFT can reduce factual hallucinations on WildHalu and Biography while largely preserving general capabilities. Recall Rewrite yields the strongest factuality gains and improves refusal behavior on UnknownBench. It thereby confirms that SFT targets beyond the base model's knowledge drive hallucination behavior.