BLiSS 1.0: Evaluating Bilingual Learner Competence in Second Language Small Language Models

AI summary

Overview

Research area: Natural Language Processing — evaluation benchmarks for cognitively inspired second language acquisition (SLA) language models, referred to in the paper as L2LMs. Technical level: Intermediate. The core idea is intuitive, but the paper relies on token-level surprisal metrics, ERRANT error annotations, and comparisons across multiple model-training paradigms. Scope in one sentence: The paper introduces BLiSS 1.0, a 136,867-triplet benchmark built from over 2.8 million naturalistic learner sentences, and uses it to show that a model's "selective tolerance" for realistic learner errors is a capability distinct from standard grammaticality.

What This Paper Is About

Existing language model benchmarks such as BLiMP and GLUE measure formal grammatical competence — whether a model prefers well-formed sentences. That is the wrong question for models meant to simulate human second language acquisition, where systematic "errors" are meaningful evidence of a learner's developing interlanguage rather than noise. The paper's goal is to build and validate a benchmark that tests whether a model finds a real learner error more plausible than a contrived error of the same type placed elsewhere in the sentence.

Key Contributions

1. A new evaluation paradigm called selective tolerance. Instead of asking only whether a model prefers grammatical over ungrammatical sentences, BLiSS asks whether a model penalizes a naturalistic human error less severely than an artificial error of the same error type at a different locus.
2. The BLiSS 1.0 dataset. 136,867 validated triplets of (corrected, learner, artificial) sentences, drawn from three English learner corpora: EFCAMDAT, the Write & Improve Corpus, and the First Certificate in English (FCE) dataset.
3. A multi-metric evaluation framework. Four metrics — Learner Preference (LP), Human vs. Artificial Preference (HAP), Robust HAP (HAP-τ), and Strict Order (SO) — all computed from token-normalized surprisal in Bits Per Token (BPT).
4. A cross-family model evaluation. Results for Bilingual LLMs, Bilingual BabyLMs, acquisition-inspired L2 models (SLABERT, B-GPT), and learner-trained models, alongside BLiMP grammaticality accuracy, showing that BLiSS scores cluster by training paradigm.

Main Findings

• Selective tolerance is distinct from grammaticality. The plot of HAP against BLiMP score shows no strong positive correlation. Models such as B-GPT reach competitive selective tolerance without top-tier grammaticality, and the paper states that high BLiMP performance does not guarantee high BLiSS performance.
• Bilingual LLMs and B-GPT are the strongest performers on the primary metrics. Both families cluster tightly with HAP scores of roughly 66–67% and the highest Strict Order scores of roughly 55–57%.
• Bilingual BabyLMs show a weaker but real signal. They perform significantly above the 50% chance baseline, but with lower Strict Order scores of roughly 35–44%.
• SLABERT and learner-trained models show a different, less successful profile. Their high Learner Preference scores (often above 50%) come with poor performance on the selective tolerance metrics, especially Strict Order, suggesting they are indiscriminately accepting of learner-like forms.
• L1-specific transfer effects are detectable. Bilingual LLMs, Bilingual BabyLMs, and B-GPT all show a statistically significant increase in performance on their own L1 data slices (p < 0.05 for the L1-versus-overall difference), which the authors take as evidence the models internalized L1-dependent transfer patterns.
• Performance clusters by training paradigm, not just by model size. Models from the same family occupy tight regions of the HAP-versus-BLiMP plot, which the authors present as validation that BLiSS captures a stable signal tied to training objectives.
• The learner-trained families vary more internally. The paper attributes this to their primary differentiator being training data rather than architecture.

Methodology in Plain English

The starting point is more than 2.8 million raw sentence-and-correction pairs from learner writing: EFCAMDAT contributed 2,711,188 pairs, Write & Improve 63,926, and FCE 52,421, for a total of 2,827,535. The team ran the ERRANT annotation toolkit over these pairs, discarded edits that were purely spelling, punctuation, or capitalization, and then "atomized" multi-error sentences so that each item contains exactly one grammatical deviation.

For every single-edit pair, a rule-based generator produced an artificial error subject to two constraints: the artificial error must involve the same grammatical operation as the human error, and it must occur at a different word position. Each candidate triplet then had to pass three validation checks — the inflected words produced must be valid English forms (verified with LemmInflect), the artificial sentence must differ from both the corrected and the learner sentence, and ERRANT must classify the artificial error as the same type as the human one. Only 4.8% of candidates survived, producing the final 136,867 triplets. A manual review of 100 triplets confirmed grammatical and positional accuracy above 95%.

Evaluation works by scoring each of the three sentences in a triplet with token-normalized surprisal in Bits Per Token, so lower BPT means the model finds the sentence more plausible. LP asks whether the learner sentence beats the corrected one. HAP asks whether the human error beats the artificial error. HAP-τ adds a buffer so the preference must exceed a small positive threshold. SO is the strictest: corrected first, learner second, artificial last. The authors recommend reporting each metric separately rather than collapsing them into one score.

Why This Matters

Impact on research. BLiSS reframes what counts as success for acquisition-inspired models. A model that merely reproduces learner errors is not necessarily learner-like; a cognitively plausible model should still prefer correct grammar while distinguishing realistic errors from contrived ones. The benchmark also gives the BabyLM-style community a shared yardstick as cognitively inspired modeling extends beyond English, and it provides a way to probe L1 transfer effects at scale using corpus metadata.

Real-world applications:

• Second language writing tools and automated feedback systems, which need to judge how plausible a specific error is in context rather than just flagging deviations.
• Language assessment and learner-corpus research, where model behavior can be analyzed across CEFR levels and L1 backgrounds.
• Adaptive tutoring or practice platforms that aim to model realistic learner output at particular proficiency stages.
• Filtering or augmentation pipelines for learner-language data, since BLiSS separates systematic error patterns from artificial ones.

Industry relevance. The benchmark is released with a HuggingFace dataset and pretrained models, and training code is open-sourced on GitHub, under a CC BY 4.0 license. That lowers the barrier for teams building writing-assistance or language-learning products to test whether their models handle learner errors in a human-like way. The paper reports support from Cambridge University Press & Assessment, indicating direct interest from the language assessment sector. The finding that selective tolerance does not track grammaticality also warns practitioners against assuming a strong general-purpose model will automatically behave sensibly on learner text.

Future Directions

• Coverage of higher proficiency levels. The dataset has only 25 triplets at C2, which the authors flag as limiting the study of advanced learner behavior and developmental trajectories.
• Discourse and pragmatic competence. BLiSS is built on sentence-level corrections and does not assess phenomena beyond sentence boundaries, leaving those dimensions unmeasured.
• Rare L1-and-error-type combinations. Because only 4.8% of candidates survived validation, infrequent combinations are represented sparsely, limiting statistical power for fine-grained analyses of those cases.
• Extending the paradigm beyond English and to more error types. The paper positions BLiSS as a research catalyst for L2 models that better reflect the diversity and systematicity of human acquisition, and notes the BabyLM Challenge's move beyond English as a context for further work.

Note: the paper does not report the specific value of the buffer τ used in HAP-τ, nor the size of the Cambridge Learner Corpus used to train the CLC models, in the content reviewed here.

Target Audience

Researchers in computational linguistics and SLA who work on cognitively inspired or learner-focused language models; developers of educational and writing-feedback tools who need to evaluate behavior on learner text; and BabyLM Challenge participants who want a benchmark that measures acquisition alignment rather than pure grammaticality. Readers should be comfortable with perplexity-style scoring and standard evaluation terminology, though the paper explains its metrics clearly enough for a motivated newcomer to follow the main argument.

Authors’ abstract

To bridge the gap between performance-oriented benchmarks and the evaluation of cognitively inspired models, we introduce BLiSS 1.0, a Benchmark of Learner Interlingual Syntactic Structure. Our benchmark operationalizes a new paradigm of selective tolerance, testing whether a model finds a naturalistic learner error more plausible than a matched, artificial error within the same sentence. Constructed from over 2.8 million naturalistic learner sentences, BLiSS provides 136,867 controlled triplets (corrected, learner, artificial) for this purpose. Experiments on a diverse suite of models demonstrate that selective tolerance is a distinct capability from standard grammaticality, with performance clustering strongly by training paradigm. This validates BLiSS as a robust tool for measuring how different training objectives impact a model's alignment with the systematic patterns of human language acquisition.