Readability $\ne$ Learnability: Rethinking the Role of Simplicity in Training Small Language Models

AI summary

Overview

Research area: Natural Language Processing — small language model (SLM) training data design, with connections to dataset curation, evaluation methodology, and the critique of anthropomorphic framing in AI research.

Technical level: Intermediate. The paper's arguments are conceptual and readable, but the evaluation setup (n-gram diversity, LLM-as-a-judge metrics, learnability ratios, constituency parsing metrics) assumes some familiarity with language modeling practice.

One-sentence scope: The paper tests whether the readability of training text (child-friendly vocabulary and syntax) or its statistical simplicity (measured by n-gram diversity) explains why very small language models trained on datasets like TinyStories generate coherent text.

What This Paper Is About

Recent work found that very small language models produce surprisingly coherent text when trained on TinyStories, a corpus of short, child-directed narratives, and this was widely interpreted as evidence that simplified, developmentally appropriate language drives capability emergence at small scale. The authors argue this interpretation conflates two distinct notions of "simple": readability for a human reader (accessible vocabulary, simple syntax) and statistical simplicity (low entropy, high redundancy, a narrow distribution over token sequences). They build synthetic datasets with matched structure but deliberately varied readability to isolate which property actually predicts coherence and learning efficiency in SLMs.

Key Contributions

1. Controlled synthetic dataset suite. The authors construct LlamaTales-Jr (a reproducible reconstruction of the TinyStories generation pipeline using open-weight Llama-3.1-8B-Instruct), LlamaTales-GRE (the same pipeline with GRE-level adult vocabulary and identical narrative feature distributions), and three domain variants (LlamaTales-History, LlamaTales-Sports, LlamaTales-News). All datasets are standardized to approximately 1 billion tokens, alongside sampled corpora FineWeb-Edu, SlimPajama, and Dolma.

2. Empirically validated measurement choices. They correlate automated readability metrics against human judgments in the CLEAR dataset, finding LLM-as-a-judge (Llama-3.1-70B-Instruct, Qwen2-72B-Instruct) reaches Pearson r = 0.74, versus r = 0.49 for Flesch-Kincaid Grade and r = 0.34 for maximum parse tree depth. They similarly validate LLM-judged coherence over perplexity as a quality proxy against a reference ranking of model capability tiers (Table A4).

3. A direct test of the readability hypothesis. Training transformers from scratch (262K to 33M non-embedding parameters, 10 billion tokens over 10 epochs) on datasets that differ in readability but match in structure, they show readability does not predict final coherence or the speed at which coherence emerges.

4. The learnability-ratio analysis. They introduce a learnability ratio (output coherence divided by training data coherence) and show it correlates inversely with 3-gram diversity, positioning statistical simplicity as the stronger predictor of learnability.

Main Findings

• Readability is not required for coherence. A 33M parameter model trained on the low-readability LlamaTales-GRE reaches in-distribution coherence comparable to Llama-3.1-70B on LlamaTales-GRE prompts, matching models trained on the high-readability LlamaTales-Jr. This pattern holds across the news, sports, and history domain variants (Figure A9).

• Readability does not speed up learning. Models trained on the less readable LlamaTales-GRE surpass a coherence score of 85 after the first epoch, while models trained on TinyStories or LlamaTales-Jr start at lower coherence and improve more gradually. If child-directed simplicity were inherently easier to learn from, the opposite ordering would be expected.

• Controlled readability differences were confirmed. Table 1 reports, for example, Flesch-Kincaid scores of 2.4 (TinyStories), 2.2 (LlamaTales-Jr), 9.6 (LlamaTales-GRE), and 10.7 (FineWeb), and LLM-judged readability of 92.6, 92.7, 64.8, and 68.2 respectively. Coherence remains high across the synthetic datasets (90.1, 89.5, and 94.4) while FineWeb is noticeably lower at 77.4, which the authors attribute to the noisier, less structured nature of web-sourced data even after filtering.

• Datasets cluster by statistical simplicity. On unique n-gram counts over 100M-token samples, synthetic datasets (TinyStories and the LlamaTales series) group separately from and below standard pretraining corpora (FineWeb, Dolma, SlimPajama).

• Statistical simplicity predicts learnability. Learnability ratios approach 1.0 for models trained on low-diversity synthetic data and drop substantially for models trained on high-diversity pretraining data. LlamaTales-GRE and FineWeb have similar readability, yet models trained on LlamaTales-GRE achieve much higher learnability, indicating readability is not the explanatory variable.

• High in-distribution coherence does not transfer. SLM outputs degrade substantially on prompts from any other dataset, including other synthetic corpora. The main exception is TinyStories and LlamaTales-Jr, which share near-identical generation processes and target audiences.

• The models are not simply memorizing. Using n-gram novelty following Merrill et al. (2024), the authors find SLMs trained on LlamaTales datasets generate substantial numbers of n-grams absent from training, particularly at 3- to 5-gram lengths, indicating recombination of learned distributional patterns rather than copying.

• Write-up numbers for two datasets differ slightly across tables. Table 1 lists LlamaTales-GRE LLM-judged readability at 64.8 and FineWeb coherence at 77.4, while Table A1 lists 72.7 and 77.6 respectively for the same datasets. The paper does not comment on this discrepancy.

Methodology in Plain English

The researchers needed to separate two things that are usually tangled together: whether text is easy for a human to read, and whether it is statistically predictable. To do this, they generated their own training corpora so they could vary one property while holding the others fixed.

They started by rebuilding the TinyStories recipe with open models instead of proprietary ones, producing LlamaTales-Jr: the same prompt template, the same curated child-friendly vocabulary list, and the same narrative features (dialogue, twist endings). Then they made a second version, LlamaTales-GRE, using the identical template and feature distribution but swapping the child-friendly word list for GRE-level adult vocabulary. Because only the word list changes, any performance difference is attributable to readability rather than structure. Three further variants re-target the same GRE-level vocabulary at different content domains.

To check the datasets actually have the properties intended, they measured readability three ways (classic formulas, constituency-parsing metrics, and LLM judgments), measured coherence with LLM judgments, and measured statistical simplicity with n-gram diversity over 1-gram through 8-gram sequences. They validated the readability and coherence measures against human judgments and a capability-tier ranking before relying on them.

Finally, they trained transformer language models from scratch on each dataset, for 10 billion tokens across 10 epochs, at sizes from 262K to 33M non-embedding parameters. They generated 1,000 completions per model-dataset pair using top-p sampling at p = 0.95 from 50-token test-split prompts, scored those completions for coherence, and compared across datasets, model sizes, and training checkpoints. Public pretrained models (GPT-2, Pythia, Mistral, Qwen2, Llama-3) served as untuned baselines.

Why This Matters

Impact on research. The paper argues that an intuitive and widely repeated explanation for TinyStories' success is not supported once readability is isolated from statistical simplicity, and that the developmental framing of small-model training risks substituting metaphor for mechanism. It connects this to broader critiques by Shanahan (2023), Placani (2024), and Ibrahim & Cheng (2025) about anthropomorphic language shaping research questions and methodology.

Real-world applications:

• Pretraining data curation for small models. Teams building on-device or low-resource models can prioritize low n-gram diversity and high structural regularity over human-facing simplicity when selecting or synthesizing data.
• Evaluation of SLM capability claims. The brittleness results show that an SLM matching a much larger model on in-distribution prompts can degrade sharply elsewhere, which affects how benchmark scores for small models should be read.
• Synthetic data pipeline design. The LlamaTales construction shows a reproducible, open-weight route to recreating a proprietary synthetic corpus with controlled vocabulary substitutions, which practitioners can reuse to build targeted domain corpora.
• Governance and public communication. The paper argues that describing models as "learning" or "developing" can lead users to overestimate understanding and shift responsibility away from those who design, train, and deploy systems.

Industry relevance. Much of the practical interest in SLMs concerns deployment where compute is constrained, and the paper's central claim is that what makes such models work is the statistical structure of the data, not the fact that it looks like children's books. That distinction directly affects data acquisition budgets, synthetic generation pipelines, and how product-facing claims about small-model capability are framed.

Future Directions

• Richer measures of dataset complexity. The authors call for going beyond n-gram diversity to capture compressibility, structural consistency, and limited long-range dependencies, since n-grams likely capture only one facet of what makes data learnable.
• Training curricula and data selection that trade simplicity against generalization. The brittleness findings raise the open question of whether statistical simplicity can be exploited without sacrificing robustness, including via mixed or broader synthetic distributions — an avenue the authors state they did not explore.
• Evaluation methods that separate genuine capability from pattern matching. The paper asks for evaluations that can disentangle real generalization from fluency enabled by statistically simple data structures.
• Testing whether the findings transfer beyond synthetic corpora. The authors explicitly state they do not attempt to generalize from synthetic to naturalistic datasets, and use FineWeb-Edu, SlimPajama, and Dolma only as higher-complexity reference points.

Target Audience

Researchers and practitioners working on small language models, pretraining data curation, synthetic data generation, and dataset design will find the core results directly actionable. The paper is also aimed at researchers studying developmental and cognitively inspired training approaches, such as the BabyLM Challenge community, since it offers an explicit counterpoint to the assumption that child-directed input is necessary for coherence at small scale. Finally, it speaks to those interested in evaluation methodology and in the critique of anthropomorphic framing in AI research and policy communication.

Authors’ abstract

Recent studies suggest that very small language models (SLMs) can generate surprisingly coherent text when trained on simplified, child-directed corpora such as TinyStories. These findings have been interpreted as evidence that readability -- characterized by accessible vocabulary, familiar narrative structure, and simple syntax -- plays a key role in enabling such capabilities to emerge. In this paper, we challenge that interpretation. We construct synthetic datasets with matched structure but varied readability, and find that readability alone does not predict coherence or learning efficiency in SLMs. Models trained on complex, adult-level text perform comparably to those trained on simplified language, and even exhibit faster development of coherence during training. Instead, we show that statistical simplicity, as measured by n-gram diversity, is a stronger predictor of learnability. Our findings caution against the growing trend of anthropomorphizing language model training -- drawing parallels to human cognitive development without empirical basis -- and argue for more precise reasoning about what properties actually support capability emergence in small models.