Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM

AI summary

Overview

Research area: Natural Language Processing — small-scale ("child-scale") language model pretraining, multilingual evaluation, and benchmark methodology.

Technical level: Intermediate. The core argument is accessible, but the paper assumes familiarity with tokenizers, minimal-pair grammaticality benchmarks (BLiMP), LoRA fine-tuning, and embedding alignment (Procrustes/Bilingual Lexicon Induction).

One-sentence scope: A French-only 125M-parameter model trained under the BabyLM 100M-word constraint is used to show that native-language benchmarks measure grammatical competence cleanly, while single-token zero-shot scoring at this scale mostly measures tokenizer and prompt-template artifacts.

What This Paper Is About

Every BabyLM submission before 2026 was trained in English, leaving open whether the lessons learned at child scale describe language acquisition broadly or just English. This paper submits the first French-only model trained from scratch under the BabyLM 100M-word budget, evaluated primarily on native Quebec-French grammatical benchmarks rather than translated English ones. Along the way, the authors discover that several standard evaluation scores at this scale are driven by the tokenizer and answer template rather than by the model's actual linguistic knowledge.

Key Contributions

1. First French-only BabyLM submission evaluated on Quebec French. MéTRON-FR is a 125M-parameter GPT-2 pretrained on 92.47M words of French, and is, to the authors' knowledge, the first model trained from scratch under the BabyLM constraint and evaluated on the Quebec dialect, using QFrBLiMP and QFrCoLA as native instruments.

2. A cross-lingual GLUE adaptation protocol with a task-type gradient. The authors combine French translation of English task data with rank-16 LoRA adapters and characterize which task types benefit (relational) versus regress (world knowledge), using a four-lever experiment grid on a single fixed checkpoint.

3. Bilingual Lexicon Induction as a child-scale geometric diagnostic. They adapt BLI to measure structural alignment between the French and English embedding spaces, showing alignment tracks acquired grammatical competence rather than training duration or data volume.

4. Forensic ablations identifying tokenizer and prompt artifacts. Tokenizer-swap tests, placebo-controlled prompting, and a lexical-frequency analysis of the translated BLiMP Supplement produce concrete reporting recommendations: tokenizer-swap sensitivity and placebo-controlled prompting should accompany headline accuracy for child-scale single-token scoring.

Main Findings

• Strong native Quebec-French grammatical competence. MéTRON-FR scores 85.97 ± 0.17% on QFrBLiMP, exceeding the best native-BLiMP scores posted in the other three official 2026 languages: English BLiMP (80.75%), Dutch BLiMP-NL (81.70%), and Chinese ZhoBLiMP (78.60%). Bucket breakdowns: syntactic 89.74%, semantic 87.19%, morphological 85.47%, anglicism-related 80.15%.

• Grammatical competence saturates before perplexity. QFrBLiMP accuracy oscillates within roughly a 0.40 percentage-point band across training epochs while training loss continues to fall monotonically from epoch 3 onward — evidence that grammatical ability and language-modeling loss decouple at this scale.

• Native versus translated benchmarks differ by 9.69 points. QFrBLiMP (85.97%, native) versus translated BLiMP (76.28%) on essentially the same kind of probe. The translated instrument conflates model competence with translation quality and lexical-frequency artifacts.

• Cross-lingual GLUE shows a task-type gradient, not uniform improvement. Relational tasks gain from French-translated task data with rank-16 LoRA: BoolQ +3.45pp, RTE +5.13pp, MRPC +2.89pp. World-knowledge MNLI regresses sharply (−11.10pp) and discourse-coreference WSC is statistically inconclusive (−1.92pp, σ = 8.88pp). The five-task mean is nearly flat (−0.31pp).

• BLI alignment tracks competence, not training duration. A frozen 768×768 orthogonal map between the French and GPT-2 embedding spaces yields p@1 = 68.84 ± 8.61% on held-out word pairs — roughly 18× above chance (3.80%). A matched-architecture English model that failed to acquire English grammar reached only 25.00% p@1, despite training on 6.50B tokens (71× more than MéTRON-FR).

• A single tokenizer swap reproduces a 7.70pp benchmark collapse. In the v1–v2–v3 ablation series, replacing the 50K Wikipedia-trained BPE tokenizer with a 16K child-directed-speech tokenizer (v3d) reproduced the entire GLUE-axiomatic drop (43.50% vs. v2's 43.30%), while the format-targeted instructions and EWoK-targeted Wikipedia did not. The authors conclude the score measures the tokenizer's vocabulary against the answer-template tokens rather than task competence.

• A nominally identical corpus rebuild shifts the score by 4.90pp. A different pre-tokenization shuffle seed reordered sentences without changing source material or word budget, yet produced GLUE-axiomatic swings comparable to differences between models. This sets an approximate 5pp training-time noise floor on single-seed claims for that benchmark.

• Placebo controls falsify the dictionary-axiom prompting effect. Apparent gains of 5–9pp from prepending a French–English vocabulary axiom block disappeared under control conditions: targeted axioms +2.33pp, random unrelated axioms +2.60pp, tautological "en=en" pairs +1.60pp. The translation-specific effect is −0.27pp. The gain is structural prompting noise, not cross-lingual lexical transfer.

• EWoK is a clean structural null at child scale. Accuracy falls inside the 95% chance interval [47.90%, 52.10%] for n = 2200 across all four interventions tested, with conditions ranging 49.86–50.95%. Inference-time tricks cannot substitute for parameters or data on world knowledge.

• LoRA preserves grammatical competence by construction. Every per-task LoRA adapter leaves the base model bit-identical, yielding an identical 83.53% on QFrBLiMP. Full fine-tuning on BoolQ drifts the score to 83.36% (−0.17pp).

• QFrCoLA acceptability judgments remain far below standard scale. Fine-tuned accuracy is 69.68 ± 0.50% with MCC 0.13 ± 0.03, compared to BERT-base on English CoLA at MCC 0.52 with roughly 27× more words. The result is positive and above chance but characterizes what is measurable at the BabyLM budget.

• Language choice is a larger axis than the architectural axis. The weighted leaderboard score of 62.80% sits at or above all re-evaluated Strict-track reference points, which the authors read as evidence that the language-choice dimension enabled by the 2025 rule change is empirically larger than the architectural innovations the field has optimized on.

Methodology in Plain English

The authors take one fixed model checkpoint and run every experiment on it, rather than selecting a different checkpoint per experiment — a discipline that keeps comparisons honest.

Training. They pretrain a 125M-parameter GPT-2 for five epochs on a French-only corpus of 92.47M words assembled from three sources: about 2.10M words of French CHILDES child-directed speech, about 24.90M words from the non-subtitle portion of BabyBabelLM's French component, and about 66M words of filtered French Wikipedia. A 50K byte-pair-encoding tokenizer is trained on the French corpus itself after comparisons against 16K, 50K, and 64K alternatives.

The Haitian-Creole oracle. Sentences are not sampled uniformly. The authors oversample French sentences dense in lemmas whose cognates survived into Haitian Creole, on the hypothesis that creoles form under communicative pressure and that surviving lemmas are disproportionately load-bearing. No Haitian Creole text enters training; the lemmas act purely as a lexical filter on French material. The authors present this explicitly as an exploratory heuristic, not a validated contribution.

Evaluation. The primary grammatical probe is QFrBLiMP, a native Quebec-French minimal-pair benchmark of 1,761 pairs across 20 linguistic phenomena, annotated by twelve native speakers and drawn from the OQLF's public normative grammar resource. QFrCoLA provides sentence-level acceptability judgments. The official BabyLM suite (BLiMP, BLiMP Supplement, GLUE, EWoK) is also run, but the English zero-shot probes are read as a lower-bound sanity check rather than a competitive target.

Cross-lingual work. Two techniques are used. Bilingual Lexicon Induction learns a frozen orthogonal map between embedding spaces on a seed dictionary of 73 bilingual lemmas (715 form-level pairs, reduced to 242 after a morphology-aware parser pass; 194 to fit, 48 held out), then measures word-translation precision at k. For downstream evaluation, rank-16 LoRA adapters are applied to French-translated task data, tested against three other levers varying task-data language, LoRA rank, and epoch count.

Ablations. Four forensic ablations isolate the sources of scoring artifacts: a tokenizer swap holding corpus, architecture, steps, optimizer, and seed constant; a placebo-controlled prompting test with targeted, random, and tautological axiom blocks; a lexical-frequency analysis of the translated BLiMP Supplement; and repeated EWoK interventions. None are pre-registered; the authors present them as constraints on interpretation.

Why This Matters

Impact on research. The paper argues that the choice of pretraining language is a larger empirical axis than the architectural innovations BabyLM has focused on, and that native-language minimal-pair benchmarks plus protocol-level diagnostics belong in the standard reporting set for child-scale submissions. Its central methodological claim is narrow but consequential: at 125M parameters and 100M words, single-token log-probability scoring on short-sequence benchmarks responds principally to the lexical-distributional properties of the tokenizer and the answer template, with task-level competence a secondary signal. Any evaluation pipeline built on that scoring mechanism without tokenizer-swap and placebo controls cannot distinguish the two.

Real-world applications:

• Benchmark design and evaluation auditing. The tokenizer-swap and placebo-controlled prompting diagnostics are directly portable to any leaderboard or internal eval harness that scores models by comparing single-token log-probabilities across answer options.

• Low-resource and non-English language technology. The recipe — modest corpus, native-language minimal-pair evaluation, competence-based rather than loss-based checkpoint selection — provides a template for languages that lack large pretraining corpora but do have native grammatical resources.

• Dialect and language-variety coverage. The Quebec-French framing shows how normative language bodies (here the OQLF) can supply authoritative evaluation instruments, and how dialect-specific findings should be scoped rather than generalized to the standard variety.

• Efficient adaptation under compute constraints. The finding that LoRA preserves the base model's grammatical competence bit-identically, while full fine-tuning degrades it slightly, is a practical argument for parameter-efficient methods when every competence parameter is hard-won.

Industry relevance. Teams fine-tuning or evaluating small models, and any organization running benchmarks that rely on single-token multiple-choice scoring, face the exact artifact this paper documents. The 4.90pp swing from a shuffle-seed change alone means that comparisons between nominally identical training runs at this scale may be measuring noise. The task-type gradient in cross-lingual adaptation — relational tasks benefit from translated task data, encyclopedic and discourse tasks do not — is actionable guidance for multilingual product teams deciding how to spend a limited annotation budget.

Future Directions

1. The missing symmetric control. The per-task gains conflate two variables: the language of the task data (English vs. French-translated) and the LoRA rank (8 vs. 16). The rank effect is isolated by a rank-16 English-LoRA condition showing no consistent gain, but the symmetric control — applying the same French-translated, rank-16 LoRA pipeline to an English-native child-scale model — was outside the compute budget. Without it, the authors cannot fully exclude that the pipeline benefits any model regardless of pretraining language.

2. Replicating the protocol cross-linguistically. The single-language design cannot adjudicate whether the native-versus-translated minimal-pair gap, the BLI competence-versus-duration contrast, and the tokenizer and placebo sensitivities are French-specific or general to child-scale non-English pretraining. Running the same protocol across the BabyBabelLM languages with native minimal-pair instruments would test whether the "right tool, right job" mapping holds cross-linguistically.

3. Isolating the Haitian-Creole oracle. The oracle's contribution is not isolated against an unweighted baseline. The clean-rebuild control bounds any oracle-specific gain to within the training-time noise floor, so the unweighted-baseline ablation remains follow-up work.

4. Making the diagnostics standard. The paper recommends tokenizer-swap sensitivity and placebo-controlled prompting as routine reporting for child-scale single-token scoring, but does not itself propose a formal standardized version of either. Turning these into reproducible, leaderboard-integrated instruments is the obvious next step.

5. Re-validating on other French varieties. The anglicism-sensitive findings do not generalize to metropolitan French without re-validation, since the metropolitan norm is more permissive on lexical anglicisms. Morphological and syntactic findings are expected to generalize but have not been tested.

Target Audience

Most valuable for: NLP researchers and engineers working on small-scale or low-resource language model pretraining, and anyone building or interpreting evaluation harnesses that use single-token multiple-choice scoring. Also relevant for multilingual product teams deciding between translation-based and native-language evaluation, for benchmark maintainers concerned about tokenizer-induced measurement error, and for computational linguists studying whether child-scale acquisition findings transfer across languages.

Less useful for: readers seeking a new architecture or a state-of-the-art result. The authors explicitly state they do not argue the submission supersedes the architectural line of BabyLM work; the contribution is a different experimental axis plus a set of methodological warnings. The paper also assumes prior familiarity with BLiMP-style benchmarks, BPE tokenization, LoRA, and Procrustes-based embedding alignment — readers without that background will need to consult the cited sources for the technical machinery.

Authors’ abstract

We submit MéTRON-FR, a 125M GPT-2 pretrained on 92.47M words of French, to the BabyLM 2026 Strict track. It scores 85.97 +/- 0.17% on QFrBLiMP (a native Quebec-French benchmark of grammatical minimal pairs) and 62.80% on the BabyLM-weighted leaderboard. A cross-lingual GLUE (General Language Understanding Evaluation) protocol that combines French task-data translation with rank-16 LoRA (Low-Rank Adaptation) produces a sharp task-type gradient: relational tasks gain measurably, while world-knowledge tasks regress. Bilingual Lexicon Induction aligns the French embeddings to GPT-2 at p@1 = 68.84 +/- 8.61%, 18X above chance, suggesting cross-lingual alignment tracks acquired grammatical competence rather than training duration. An ablation study shows that single-token zero-shot scoring is dominated by tokenizer and template artifacts at the child scale, motivating tokenizer-swap sensitivity, placebo-controlled prompting, and native-language minimal-pair benchmarks as standard diagnostics.