Procedural Pretraining: Warming Up Language Models with Abstract Data

AI summary

Overview

Research area: Natural language processing; specifically language model pretraining data composition and the transfer of abstract algorithmic skills to semantic domains.

Technical level: Intermediate. The paper uses standard transformer pretraining and fine-tuning setups, but its arguments rest on familiarity with pretraining paradigms, formal languages, and layer-wise interpretation of transformer components (attention vs. MLP).

Scope in one sentence: The paper studies whether briefly pretraining language models on abstract, algorithmically generated "procedural" data—formally, before standard pretraining on natural language, code, or mathematics—improves and accelerates the subsequent learning of semantically rich data.

What This Paper Is About

Large language models learn semantic knowledge and abstract reasoning skills at the same time, in a single entangled process. The authors ask whether separating these two things helps: what if a model first sees simple, non-semantic data produced by algorithms (such as balanced brackets, sorting, or cellular automata) and only afterwards begins standard pretraining on real text, code, or informal math? The goal is to test whether this cheap "warm-up" stage improves both the speed and the final quality of ordinary pretraining, and to understand where in the model the benefit is stored.

Key Contributions

1. Probing procedural pretraining with algorithmic tasks. The authors show that different forms of procedural data each strengthen specific algorithmic skills, that the learned information is localised in specific layers (attention vs. MLPs), and that simple explanations such as weight rescaling or generic attention sharpening do not account for the improvements.

2. Transfer to pretraining on diverse semantic domains. They show that gains on algorithmic skills carry over to natural language, code, and informal mathematics, with as little as 0.1–0.3% extra procedural tokens, and that procedural data can also substitute for a substantial fraction of standard data.

3. Localising the transferable pretrained information. They identify which model components carry the useful information per domain: attention layers are more important for structured, language-free domains such as pure code, while MLP layers help natural language more.

4. Combining multiple forms of procedural data. They explore two techniques—mixing data types during pretraining, and surgically assembling weights from separately pretrained models—and report promising results as a proof of concept.

Main Findings

• Retrieval improves dramatically from a single procedural data type. Accuracy on the needle-in-a-haystack context recall task jumps from 10 to 98% when the model is pretrained on Dyck sequences (balanced brackets).

• Different procedural data types help different skills. Pretraining on k-Dyck improves Haystack, while Eca rule 110 benefits Reversed Addition; the best-performing procedural data type varies by target task.

• The structure in the data is essential, not the token distribution. When the best-performing procedural sequences are randomly shuffled—preserving the token distribution but destroying the structure—performance drops back to the baseline.

• Selective layer transfer can outperform full-model transfer. With the Identity/Haystack pair, attention-only transfer gives an 80-percentage-point improvement over full-model transfer, indicating that useful information lives in the attention layers while other pretrained components hold non-transferable structure. An exception is Reversed Addition, where MLP-only and full-model transfer are superior.

• Simple explanations are ruled out. An explicit regularizer for sharper attention does not replicate the benefits, and shuffling weights per layer (preserving magnitude distributions but erasing structure) causes accuracy to drop dramatically, as does adding Gaussian noise to the weights.

• Tiny amounts of procedural data improve standard pretraining. Front-loading 0.1–0.3% procedural data significantly outperforms standard pretraining on C4, CodeParrot, and DeepMind-Math; models reach the same loss with only 55%, 67%, and 86% of the original data respectively, implying comparable FLOPs reductions.

• Procedural tokens efficiently substitute for semantic tokens. On C4, about 45% of semantic tokens (and FLOPs) can be saved while matching baseline loss by using only 2.1M procedural tokens; for CodeParrot and DeepMind-Math the savings are 33% and 14% using 2.1M and 10.4M procedural tokens respectively.

• Benefits survive downstream fine-tuning. Improvements persistently carry over to WikiText-103, GLUE, PY150, MBPP, ARC-Easy, and HellaSwag. A 124M CodeParrot model with procedural pretraining begins to show non-random accuracy on MBPP, whereas the baseline shows zero accuracy up to 80 attempts.

• Localisation depends on the domain. Attention layers matter most for pure code (JavaCorpus), MLPs matter most for natural language (WikiText, C4), and full-model transfer is best for domains mixing language with structured data (documented code, informal mathematics). In a substitutive setting, MLP-only transfer needed only 42% of C4 tokens to match the perplexity level that full-model transfer reached with 55%.

• Procedural data types combine additively. Mixtures of Set and Union with various ratios outperform single-source baselines, and a model assembled from Set-pretrained attention layers plus ECA Rule 110-pretrained MLPs (average 89.3 across four tasks) outperformed all single-source configurations, which each showed weaknesses on at least one task.

Methodology in Plain English

The authors train GPT-2-style decoder-only transformers from scratch with a standard next-token prediction objective. Before any normal training, the model is exposed to "procedural data," which is generated by explicit algorithms rather than by a trained model. The procedural sources include sequence transformations (Set, Reverse, Identity, Union, Sort, Delete), a new stack-memory simulation (Stack), formal languages for balanced parentheses (k-Dyck and k-Dyck Shuffle), and the elementary cellular automaton Eca rule 110. Sequences are short, at most 128 tokens, and when input/output pairs are used the loss is computed only on output tokens.

Two experimental settings organise the work. In the additive setting, the amount of standard data is held fixed and procedural tokens are added, measuring whether procedural data supplies a training signal that semantic data alone does not. In the substitutive setting, standard tokens are reduced while procedural tokens increase, measuring how much standard data procedural data can replace at matched performance. Two transfer modes are compared: full-model transfer (all procedurally pretrained weights are kept) and selective transfer (only attention or only MLP layers are kept, with the rest reinitialised).

For diagnostics, small two-layer, four-attention-head transformers are pretrained on one procedural type and then fine-tuned on algorithmic tasks—needle-in-a-haystack with 30 key-value pairs, addition of two 5-digit integers, reversed addition of 10-digit numbers, multiplication of two 5-digit integers, and sorting 10 integers from 0 to 99. Each of these experiments is run with 10 different seeds. For the scaling study, CodeParrot-small-style models are used with 0 to 20M procedural tokens followed by 655M, 1B, or 1.6B standard tokens from one of the target datasets, with models scaling up to 1.3B parameters and data up to 10.5B tokens in the contribution summary (Appendix L describes scaling to 10B tokens).

Why This Matters

Impact on research. The work reframes procedural data from a substitute for standard pretraining into a complementary warm-up stage, and it provides a layer-level account of where the benefit is stored. This supports the view that knowledge acquisition and reasoning skill acquisition can be partially disentangled, and it gives a controlled setting for studying why structured pretraining data—such as computer code—is empirically effective.

Real-world applications.

• Reducing pretraining cost by reaching a target loss with less semantic data and fewer FLOPs.
• Improving long-context retrieval, since context recall rose from 10 to 98% in the diagnostic setting.
• Improving code generation and completion, as models showed non-random MBPP accuracy where the baseline scored zero.
• Strengthening language and commonsense downstream tasks (GLUE, HellaSwag, ARC-Easy) at only a trivial extra compute cost.

Industry relevance. Frontier pretraining is expensive, and even a 14–45% reduction in tokens to reach a given loss, achieved with 0.1–0.3% extra procedural tokens, translates to meaningful compute savings. The layer-selective transfer results also suggest that practitioners can transfer only the model components that matter for their target domain rather than entire checkpoints.

Future Directions

• Scaling up. The authors note that their models go up to 1.3B parameters and use a lower data-to-model ratio than state-of-the-art LLMs, so further scaling is identified as an important next step.

• Efficient initialisation. Because procedural data has small Kolmogorov complexity and could be summarised in a few lines of code, the authors suggest it may be possible to replace procedural pretraining with a deterministic or closed-form "smart initialisation."

• First-principles explanation and mechanistic interpretability. Since simple explanations were ruled out, the authors propose investigating the mechanisms at play using mechanistic interpretability techniques.

• Optimised combinations of procedural data. The combination experiments are described as a proof of concept; the authors suggest adapting existing data-mixture optimisation methods to balance multiple types of procedural data, and exploring larger, more optimised combinations.

Reported limitations. The paper explicitly lists two: the use of smaller models and lower data-model ratios relative to state-of-the-art LLMs, and the preliminary nature of the multi-type procedural data combination experiments.

Target Audience

Researchers and practitioners working on language model pretraining, data curation, and training efficiency will benefit most. It is also relevant to those studying the separation of knowledge and reasoning in LLMs, the transferability of inductive biases, and the interpretation of attention versus MLP layers. Readers looking for an accessible entry point into why structured data such as code helps LLMs will find the diagnostic section on algorithmic tasks particularly useful.

Authors’ abstract

Pretraining language models directly on web-scale corpora is the de facto paradigm. We study an alternative where the model is initially exposed to abstract structured data to ease the subsequent acquisition of rich semantic knowledge, much like humans learning simple logic and mathematics before higher reasoning. We focus on procedural data, generated by formal languages and other simple algorithms, as such abstract data. We first diagnose the algorithmic skills that different forms of procedural data can improve, often significantly. For example, the accuracy of context recall (Needle-in-a-haystack) jumps from 10 to 98% when a model is pretrained on Dyck sequences (balanced brackets). Second, we study how these gains are reflected in pretraining larger models (up to 1.3B). We find that front-loading as little as 0.1 to 0.3% procedural data significantly outperforms standard pretraining on natural language, code, and informal mathematics (C4, CodeParrot, and DeepMind-Math datasets). Notably, this also enables the models to reach the same loss value with only 55/67/86% of the original data and thus a comparable reduction in FLOPs. Third, we explore the mechanisms behind the benefits and find that procedural pretraining instills non-trivial structure in both attention and MLP layers. The former is particularly important for structured domains (e.g. code), and the latter for language. Finally, we lay a path for combining multiple forms of procedural data. Our results show that procedural pretraining is a simple, lightweight means of improving performance and accelerating language model pretraining, ultimately suggesting the promise of disentangling knowledge acquisition from reasoning in LLMs.