Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Setting

AI summary

Overview

Research area: Natural Language Processing — linguistic evaluation of large language models, specifically grammatical competence benchmarking for a low-resource, endangered language (Irish).

Technical level: Intermediate. The paper is readable without deep linguistics training, but it assumes familiarity with concepts such as minimal pairs, log-likelihood scoring, prompting strategies, and acceptability judgements.

Scope: The paper introduces Irish-BLiMP, a handcrafted benchmark of 1,020 Irish minimal pairs spanning 11 linguistic features, and uses it to compare ten LLMs against fluent human speakers.

What This Paper Is About

Existing benchmarks for testing whether language models actually understand grammar have focused mainly on majority languages like English and Chinese. Irish is an endangered, extremely low-resource language, and prior multilingual benchmark work covers it with only 28 minimal pairs, which is too few to draw meaningful conclusions. This paper builds a much larger, expert-reviewed Irish benchmark and uses it to measure how far current LLMs lag behind fluent human speakers on Irish syntax and morphology.

Key Contributions

1. A new dataset and framework. Irish-BLiMP (Irish Benchmark of Linguistic Minimal Pairs) is presented as the first dataset and framework for fine-grained evaluation of linguistic competence in Irish, comprising 1,020 manually constructed minimal pairs across a taxonomy of 11 linguistic features.

2. A measured open- versus closed-source gap. Extensive experiments reveal a substantial performance gap between state-of-the-art open-source and closed-source models: a performance gap of 18.1% persists between them, with open-source models performing near the random baseline while the strongest closed-source model (gpt-5) reaches only 73.5% accuracy.

3. A human–LLM comparative assessment. Three native Irish speakers produced 3,060 annotations with an average accuracy of 90.09% (agreement rate 77.27%), showing the benchmark is straightforward for humans yet challenging for LLMs.

4. Evidence about model representation. The contrast between human and model error patterns suggests current models rely heavily on surface pattern recognition rather than an internalised understanding of Irish grammatical rules.

Main Findings

• Humans lead everywhere. Human participants outperform all models across all 11 linguistic features, ranking first in every category, with 16.6% higher accuracy on average. The top models, gpt-5 and gemini-2.5-flash-lite, reached only 73.5% and 67.1% respectively, against 90.1% for humans.

• Clear closed-source versus open-source divide. Closed-source models (gpt-5, claude-sonnet-4.5, gemini-2.5-flash-lite) averaged 68.8% compared to 50.7% for open models. All open-source models perform near random chance (approximately 50.0%).

• Size does not guarantee accuracy. gemma-3-27b-it (27B parameters) outperforms much larger models such as Llama-4-Scout-17B-16E-Instr. (109B parameters) and gpt-oss-120b (120B parameters) in overall accuracy, but this advantage is inconsistent: it underperforms by up to 20.0% on "Adjectives and Comparison" compared to Llama-4-Scout, while surpassing larger models by over 10 points on "Clause Structure and Word Order".

• Irish-specific fine-tuning does not help. The Irish-adapted UCCIX model slightly underperforms its base model (Llama-2-13B) by 0.2% average accuracy, suggesting fine-tuning alone may be insufficient when the available Irish corpus is small.

• Models and humans struggle with different things. All LLMs show weak correlation with human performance (−0.7 < r < 0.7); the strongest alignment is claude-sonnet-4.5 at r = 0.47. Fewer than half of the ten evaluated LLMs exhibit strong inter-model correlation.

• Model accuracy is far more erratic across categories. Human performance shows low variance (σ = 4.29), while LLM variance reaches 12.24 for claude-sonnet-4.5, with other models ranging from 5.43 (Mistral-Small-3.2-24B-Instr.) to 8.55 (gemini-2.5-flash-lite).

• Extra context helps a little, not enough. Few-shot and grammar-book context improve most models, though neither consistently dominates, and some families (gpt-oss, Llama-2) show no clear difference over 0-shot. Llama-4 improves from 47.2% (0-shot) to 68.7% (5-shot). The best absolute performance is gpt-5 under 5-shot prompting at 81.1%, still below the human baseline (88.0%) even though models received more contextual information than human evaluators.

• A failure mode with added context. claude-sonnet-4.5, when given additional context, generates extended explanations instead of the required letter label ("A" or "B"), causing exact-match failures.

Methodology in Plain English

The team consulted Irish grammar reference works and linguistic literature to define a taxonomy of linguistic phenomena organised at three levels: type, phenomenon, and paradigm. Three fluent Irish speakers then manually wrote ten minimal pairs for each feature — 1,020 pairs in total across 102 paradigms. A minimal pair is two sentences that differ in only one targeted way, where one is grammatical and one is not (for example, "Labhróidh Liam amárach" versus the ungrammatical "Liam labhróidh amárach", testing Irish verb–subject–object order). No LLM was used to generate any of the pairs, in contrast to template-based or model-generated approaches. Each pair was reviewed by at least two fluent participants. The judgements follow the standard written form of Irish, An Caighdeán Oifigiúil, and do not cover dialect-specific features.

For evaluation, the task was framed as a multiple choice between the acceptable and unacceptable sentence. Open-source models were scored by comparing the output probability (log-likelihood) of each option, while closed-source API models were prompted and their textual replies matched against the correct label. Temperature was set to 0, reasoning effort was minimised or disabled for reasoning-capable models (gpt-5, gpt-oss-120b, gpt-oss-20b), and exact label matching was required. The ten evaluated models were gpt-5, claude-sonnet-4.5, gemini-2.5-flash-lite, gpt-oss-120b, gpt-oss-20b, Llama-4-Scout-Instruct, gemma-3-27b-it, Mistral-Small-3.2-24B-Instruct, UCCIX, and Llama-2-13B. Three prompting strategies were tested: zero-shot (directly comparable to the human setup), few-shot with five additional example pairs, and grammar-book context providing a brief description of each paradigm alongside the pair.

Why This Matters

Impact on research. Irish-BLiMP offers the first systematic framework for assessing grammatical competence in Irish and a discriminative alternative to aggregate metrics like BLEU, COMET and GLUE/SuperGLUE, which reveal overall performance but not which specific syntactic phenomena a model handles well or badly. It also documents a gap in multilingual generalisation for an extremely low-resource, endangered language.

Real-world applications:

• Informing development of Irish-language tools such as translation, educational software, and public-service interfaces.
• Supporting digitalisation and preservation efforts for endangered languages.
• Providing a template that can be adapted to build equivalent benchmarks for other low-resource languages.
• Guiding organisations that deploy LLM-powered services in bilingual or minority-language contexts, where limited compatibility with minority languages can exacerbate technological language inequality.

Industry relevance. The paper shows that closed-source models retain a substantial advantage (68.8% vs 50.7% on average) over open-weight alternatives for Irish, and that neither scale alone nor language-specific fine-tuning closes the gap. Organisations building multilingual products need to know that frontier models may still be far behind human speakers on underrepresented languages, and that a fine-tuned small model may not beat its base model.

Future Directions

• Expanding the dataset's coverage, for example to dialectal variations of Irish; the current benchmark is limited to written standard Irish and does not fully capture dialectal diversity or phonologically conditioned variation.
• Extending the framework to other languages, since the creation and evaluation methodology is described as extensible in principle.
• Investigating why language-specific fine-tuning did not yield gains, and how to achieve more effective multilingual transfer for extremely low-resource languages with small corpora.
• Determining what training or architectural changes would produce error patterns that align more closely with human speakers, given the weak human–model correlation (strongest at r = 0.47).

Target Audience

Researchers working on low-resource and endangered language NLP; linguists interested in acceptability judgements and syntactic variation; practitioners evaluating or fine-tuning multilingual LLMs; and Irish-language technology developers and language-preservation bodies. It is also useful for readers interested in how benchmark design differs between high-resource and low-resource settings.

Authors’ abstract

We present Irish-BLiMP (Irish Benchmark of Linguistic Minimal Pairs), the first dataset and framework designed for fine-grained evaluation of linguistic competence in the Irish language, an endangered language. Drawing on a variety of linguistic literature and grammar reference works, we manually constructed and reviewed 1020 minimal pairs across a taxonomy of 11 linguistic features, through a team of fluent Irish speakers. We evaluate both existing Large Language Models (LLMs) and fluent human participants on their syntactic knowledge of Irish. Our findings show that humans outperform all models across all linguistic features, achieving 16.6% higher accuracy on average. Moreover, a substantial performance gap of 18.1% persists between open- and closed-source LLMs, with even the strongest model (gpt-5) reaching only 73.5% accuracy compared to 90.1% by human. Interestingly, human participants and models struggle on different aspects of Irish grammar, thus highlighting a difference in representation learned by the models. Overall, Irish-BLiMP provides the first systematic framework for evaluating the grammatical competence of LLMs in Irish and offers a valuable benchmark for advancing research on linguistic understanding in low-resource languages.