Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views

AI summary

Overview

Research area: Natural language processing, specifically how large language models acquire knowledge during pre-training and continued pre-training, and how that knowledge should be represented in text.

Technical level: Intermediate. The core ideas are intuitive, but the paper relies on controlled pre-training experiments, probe-based evaluation, and an MLP channel-level mechanistic analysis.

Scope (one sentence): The paper runs controlled continued-pre-training experiments on 36 domain documents across computer science, law, and medicine to test whether "auxiliary views" — blogs, textbooks, and Stack Exchange-style Q&A reformulating the same knowledge — improve knowledge acquisition over plain repetition and paraphrasing.

What This Paper Is About

Most work on pre-training data focuses on corpus-level properties such as deduplication, filtering, quality, and diversity, but leaves open the question of how knowledge should actually be represented in text. The authors argue that the same knowledge naturally appears in many forms — tutorials, blogs, forum Q&A — which they call auxiliary views, and they test whether these views causally help a model learn. Their goal is to isolate this effect with controlled experiments on complex, interdependent domain knowledge rather than the biographical facts used in prior work.

Key Contributions

1. Isolating auxiliary views as a distinct factor. The authors define auxiliary views as human-style reformulations of knowledge that go beyond paraphrase — different genres, contexts, and framings — and construct a controlled experimental setup that holds the token budget, data ordering, and all other factors constant while varying only the representation of the knowledge.

2. Demonstrating that reallocating repetition tokens to auxiliary views improves both understanding and factual recall. Counterintuitively, this holds even for factual probes whose target spans are verbatim phrases from the source document, which the model then sees less often.

3. Showing the effect is not distillation from a strong teacher. Across eleven generator configurations spanning multiple model families, reasoning-effort levels, and sizes, downstream factual accuracy is stable and uncorrelated with generator size or the generator's own factual accuracy.

4. Providing mechanistic signatures via FFN channel analysis. Auxiliary views reduce the magnitude of parameter movement relative to paraphrasing while redistributing where learning occurs — more change in the middle and final layers, less in an upper-middle band around layers 16–24.

Main Findings

• Auxiliary views beat both source repetition and paraphrase. On OLMo-2-32B, the ordering Para. 9 + Aux. > Para. 9 > Source holds across log probability, multiple-choice accuracy, and target rank, is established early in training, and persists until convergence. Source learns faster on factual probes for roughly the first 20 steps.

• The advantage emerges with scale. The 1B model derives little advantage from auxiliary views, while the gap over Source and Para. 9 widens steadily through 7B, 13B, and 32B, a trend largely consistent across domains.

• Factual recall improves even though target spans appear less often. In the lexical-bias check, auxiliary views contain full factual target spans at frequency 0.112 per 1k tokens with coverage 0.33, versus 0.391 / 0.70 for the source and 0.222 / 0.60 for the 49 paraphrases. Auxiliary views perform better despite stating the targets less frequently.

• The advantage survives reduced upsampling. When matching upsampling is halved (0.5 regime), peak factual MCQA is 0.412 for auxiliary views, 0.399 for Para. 9, and 0.382 for Source. With no upsampling, it is 0.392, 0.389, and 0.380 respectively. Source never improves as duplication is reduced.

• Results generalize to a pre-training-faithful setting. Resuming OLMo-2 7B from step 925,000 with its optimizer state, original data stream and schedule, and a global batch size of 1,024 yields factual MCQA of 0.403 and inference MCQA of 0.492 for auxiliary views, versus 0.372 / 0.421 for token-matched Source and 0.375 / 0.417 for token-matched Para. 9.

• Human-written views and another model family behave similarly. An experiment with human auxiliary views collected from the open web covers only two documents and is treated as suggestive, but the benefit again grows with model size. The main experiment on Qwen-2.5-7B shows auxiliary views again yielding the largest gains.

• Which view type matters little. Textbooks, blogs, and Stack Exchange Q&A all perform similarly, with a slight benefit from mixing view types on factual recall.

• Mechanistic compression. Paraphrasing induces the largest parameter movement of the three conditions in both norm and cosine distance, while adding auxiliary views reduces that magnitude, though it remains above source-only training. All conditions concentrate change in the middle and final layers with a dip in the upper-middle layers; auxiliary views accentuate this, changing middle and final layers more and the upper-middle band (layers approximately 16–24) less. At 1B this per-channel signature is almost uniformly negative or zero, lacking the layer-wise bias seen at 7B.

• Teacher strength is irrelevant, view text volume is not. Downstream factual accuracy across eleven generator configurations ranges from 0.405 to 0.422, all above the Para. 9 baseline of 0.396. It is uncorrelated with generator size (Pearson r = −0.14, n = 6) and with the generator's own factual accuracy (r = −0.24, n = 10, p = 0.50). Scaling within a family goes the wrong way (gpt-oss-20B to 120B: 0.422 to 0.413; Gemma-4 12B to 31B: 0.414 to 0.405). The volume of generated view text, however, does correlate with downstream accuracy (r = +0.62, p ≈ 0.04). Notably, gpt-oss-20B has the least measured domain knowledge yet its views teach best.

• Paraphrasing helps only under specific conditions. Source saturates after roughly 20 exposures and then sharply degrades as the model overfits, while Para. 9 sustains improvement through roughly 40 exposures. This benefit emerges only at 7B and above; at 1B paraphrasing is slightly harmful. The advantage depends on batch size: Source becomes more stable as batch size grows and nearly matches Para. 9 by batch size 256, while Para. 9 stays roughly flat. For factual acquisition at small batch sizes, paraphrasing improves learning at 7B and above, but the advantage shrinks at larger batches and reverses at batch size 256 for 7B and 13B.

• This reconciles conflicting prior results. Chang et al. (2024), at batch size 2048 with 2048-token chunks, reports degraded factual learning from paraphrasing, whereas Allen-Zhu and Li (2024), at batch size 96 with 512-token chunks, reports gains.

• A prior-knowledge gap exists and can be narrowed. Prerequisite-topic MCQA accuracy rises from 0.4380 to 0.5454 for OLMo-2-0425-1B and from 0.6859 to 0.7272 for OLMo-2-1124-7B after continued pre-training on synthetic textbooks covering the foundational material. Relative to standard Para. 9, providing contextual or prerequisite knowledge substantially improves acquisition, but under the stricter token-matched comparison it does not consistently surpass Para. 9. Contextual knowledge drives larger factual gains, prerequisite knowledge yields greater inference improvements.

• Where prior knowledge is placed matters little. Varying whether prerequisite data appears at the beginning, middle, or end of training produces no consistently best placement across metrics, and differences are small.

• Learning rate may have limited the measured effects. The authors use a fixed peak learning rate of 4e-5, while the 13B and 32B models were pre-trained at higher rates (9e-5 and 6e-5) than the 7B model (3e-5). The advantages of auxiliary views and paraphrasing over source-only training widen as learning rate increases.

Methodology in Plain English

The authors assemble 36 self-contained documents to serve as the target knowledge: twelve recent arXiv computer science papers, twelve U.S. federal appellate legal opinions, and twelve PubMed Central medical case reports. They check all 36 against the Infini-gram API using OLMo-2's pre-training index and find zero matches, so the knowledge is genuinely new to the model.

For each document they generate several kinds of text. Paraphrases (49 per document, 1,764 total) are produced by GPT-4.1 and serve as the linguistic-variation control. Auxiliary views — textbooks, Stack Exchange-style Q&A, and blogs (Table 1 uses Direct Preference Optimization material as the worked example) — and prerequisite textbooks are produced by GPT-5-mini. Contextual knowledge is collected as cited arXiv papers and cited judicial opinions; medical case reports lack an analogous citation structure, so that domain is omitted for contextual views.

Learning is measured with LAMA-style cloze probes. Factual probes extract knowledge-bearing sentences and convert them into self-contained statements with the answer span at the end; inference probes require combining facts that are never stated together. The pipeline yields 6,435 factual and 430 inference probes, plus multiple-choice variants (4,515 factual and 322 inference MCQs), with 200 probes of each type manually validated.

Training uses single-batch knowledge injection on OLMo-2 base models (1B, 7B, 13B, 32B): the documents fit in one forward pass, the rest of the batch is filled with general data streamed from the DCLM subset, and N = 100 injections are performed. Three conditions are compared — Source (original document injected every batch), Para. M (cycling through the original and its M paraphrases), and Para. M + Aux. (auxiliary views injected alongside paraphrased documents). All conditions are token-matched by upsampling Source and Para. M so every condition sees the same number of knowledge-bearing tokens, meaning Para. M + Aux. devotes fewer tokens to direct repetitions. Evaluation uses joint log-probability of the target span, target rank under teacher forcing, and 5-shot multiple-choice accuracy with decoding constrained to the answer choices.

For the mechanistic analysis, the authors compare trained FFN weights against the base model, measuring relative delta norm, cosine distance, and the Gini coefficient of change across channels, per layer and over training. They focus on MLP layers because they hold most of a transformer's parameters and because individual channels have been shown to act as key–value memories.

Why This Matters

The paper reframes data diversity from a vague corpus-level property into something that can be constructed deliberately around individual pieces of knowledge. It offers an operational account of why diverse data helps, and it suggests a practical route for making continued pre-training more reliable in specialized domains — a setting where a 70B model continually pre-trained on Wiki-style documents reportedly recalls only 62.7% of facts despite sophisticated augmentation.

Real-world applications:

• Scientific and technical domain adaptation. Open scientific corpora constantly produce new research that lacks the surrounding explanatory material — tutorials, forums, Q&A — that established knowledge accumulates. Synthesizing auxiliary views could help models absorb emerging findings.
• Clinical and biomedical continued pre-training. Prerequisite textbooks can close foundational gaps before target material is introduced, which the authors recommend as a first practical step.
• Legal and medical document processing. Both domains are covered in the experiments, and legal citations provide natural contextual knowledge.
• Data-scarce synthetic pre-training pipelines. The results offer guidance on when paraphrasing is worth generating and when the token budget is better spent on conceptually distinct reformulations.

Industry relevance: The findings speak directly to how labs construct synthetic pre-training corpora. Two conclusions are actionable: generator quality is not the bottleneck (a weak generator's views taught as well as a strong one's), and generating more view text does correlate with better downstream accuracy. The batch-size dependence of paraphrasing also means the value of a data-augmentation strategy depends on the training configuration it is used in.

Future Directions

• Whether synthetic auxiliary views work for long-tailed specialized knowledge. The authors explicitly flag as open whether the approach remains effective in domains where the LLM itself may lack the expertise to generate high-quality views.
• Whether teacher strength ever matters. The generator-strength result may not generalize to low-resource, highly specialized domains where even comprehending the source material may challenge the generator.
• Extending beyond the studied scale. All results are constrained to models up to 32B, and the authors note that findings may not extend to substantially larger models.
• Broader curriculum and domain coverage. Only three domains are covered, and the contextual-knowledge results were not consistent across them; the authors leave a fuller investigation of curriculum effects (ordering of prior knowledge) to future work. A more thorough study of how each auxiliary view type facilitates learning differently is also called for.

Target Audience

Researchers and engineers working on pre-training data curation, synthetic data generation, and continued pre-training for domain adaptation — particularly those in scientific, biomedical, or legal settings. It is also relevant to readers interested in the mechanistic side of knowledge storage in transformer MLP layers, since the paper connects a data-level intervention to a specific parameter-change pattern. Readers seeking a purely conceptual overview without experimental detail will find the framing and practical takeaways accessible, while the probe design, token-matching controls, and FFN analysis reward a more technical readership.

Authors’ abstract

Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training. We posit that auxiliary views, reformulations of knowledge, are causally helpful for learning. We design controlled experiments to isolate this. First, we confirm that repetition is necessary for acquisition and clarify that paraphrasing helps only at smaller batch sizes. Second, holding the token budget fixed, allocating tokens from document repetition to auxiliary views improves learning, counterintuitively, even for factual recall. Third, the effectiveness of auxiliary views is not contingent on the strength of the teacher model that generates them. Fourth, we identify forms of knowledge, contextual and foundational, that aid learning in the presence of prior knowledge gaps. Finally, we examine how these effects manifest mechanistically via layer-wise biases and compression. Together, our findings suggest that auxiliary representations of knowledge, which arise naturally in large pre-training corpora, are a key factor in the success of pre-training and offer a plausible explanation for why data diversity matters.