LingGym: How Far Are LLMs from Thinking Like Field Linguists?

AI summary

Overview

Research area: Natural Language Processing / Computational Linguistics — specifically evaluation of large language models on meta-linguistic reasoning over low-resource and endangered languages.

Technical level: Intermediate. The paper assumes familiarity with LLM benchmarking and with basic field-linguistics notation (interlinear glossed text), but its core idea is accessible.

Scope (one sentence): The paper introduces LingGym, a benchmark built from 18 reference grammars that tests whether LLMs can infer a missing word or gloss from structured linguistic data (segmented sentences, glosses, grammatical explanations, and translations), and it reports accuracy for ten publicly available models across four levels of provided context.

What This Paper Is About

Field linguists describe unfamiliar, often endangered languages by working through structured materials: interlinear glossed text (IGT), where each morpheme is aligned with a grammatical gloss and a free translation, plus prose explanations of grammatical rules. The paper asks whether LLMs can actually reason over this kind of expert-curated meta-linguistic information, rather than memorizing or pattern-matching on raw text. To answer this, the authors build LingGym and a controlled multiple-choice task in which one word and its gloss are masked, and the model must recover them using varying amounts of linguistic context.

Key Contributions

1. A cleaned, structured IGT dataset. The authors extract IGT examples from 18 publicly available reference grammars published by Language Science Press (from the Studies in Diversity Linguistics and Comprehensive Grammar Library series), convert the LaTeX sources to plain text, filter chapters by linguistic subfield, and hand-verify alignment between words, morphemes, glosses, and translations. The result is 19,612 IGT examples linked to knowledge points (KPs) across 8 language families.

2. An evaluation framework grounded in descriptive linguistic resources. The paper defines the Word-Gloss Inference task: a cloze-style multiple-choice question with four options, where a single word (and its gloss) is masked and must be inferred from the sentence, the gloss line, a grammatical explanation, and/or an English translation.

3. A principled distractor-generation scheme. Three strategies generate plausible wrong answers: a form-based (longest-common-substring) distractor, a semantics-based distractor using Sentence-BERT embeddings, and a chapter-local distractor sampled from the same grammar chapter. The correct answer is randomly placed in positions A–D so each position holds it approximately 25% of the time.

4. A benchmark and typologically informed analysis of state-of-the-art LLMs. Ten models from Qwen2.5, Gemma 3, DeepSeek-R1, LLaMA3, and GPT-4 (o4-mini) are evaluated across four difficulty settings, with per-language, per-family, per-subfield, chain-of-thought, and error analyses. The benchmark is released on GitHub.

Main Findings

• Even the raw sentence alone shows above-chance performance, suggesting some data contamination. Since all grammar sources are openly available as LaTeX on GitHub, the authors test with only the original sentence (setting S). All models exceed the 25% chance level, and larger models score higher, which the authors read as evidence of memorization. Overall performance remains far from perfect, so they treat LingGym as a meaningfully challenging benchmark. Scores on S span 29.62 to 43.63 across the ten configurations reported in the main table.

• Adding structured linguistic information improves performance consistently. Accuracy rises monotonically from S to S+G to S+G+KP to S+G+KP+T for every model family and size in the main table. On the full S+G+KP+T setting, the reported range is 50.32 to 81.17, and the best-performing model, DeepSeek-R1 32B, reaches around 81% accuracy.

• Glosses, translations, and knowledge points each contribute independently. In the permutation ablation on four selected models, every combination that adds information outperforms the sentence-only condition. For example, Qwen2.5-7B moves from 33.04 (S) to 48.65 (S+T), 52.78 (S+KP), 66.59 (S+G+T), and 71.09 (S+G+KP+T); LLaMA3-8B moves from 29.62 (S) to 50.32 (S+G+KP+T). No model reaches perfect accuracy, which the authors take as evidence of substantial remaining headroom.

• Larger models clearly outperform smaller ones. Within each family the larger variant wins on the main table: Qwen2.5-32B over 7B, Gemma3-27B over 12B and 4B, DeepSeek-R1-32B over 7B, and LLaMA3-70B over 8B.

• Chain-of-thought prompting does not clearly help. Figure 5 shows no conclusive evidence that CoT prompting improves meta-linguistic reasoning across any of the model families tested. Reasoning models are also not uniformly advantaged: DeepSeek-R1 32B dominates in almost all conditions, but DeepSeek-R1 7B shows no such advantage, and o4-mini is not competitive with the strongest non-reasoning models.

• Performance is relatively stable across languages, families, and subfields. The per-language breakdown and the weighted-average plots show only minor variation, which the authors interpret as evidence that the benchmark is representative and balanced within and across languages, and that LLMs reason to some extent across linguistic subfields.

• Three dominant error types. Analysis of the strongest configuration (DeepSeek-R1 32B at S+G+KP+T) yields: (i) abbreviation-heavy items with opaque gloss tags, where models treat dense tags as uninterpreted symbols — e.g., preferring DEM=Pl over bare DEM in Moloko, or zane (DEM:PROX) over zane=me (DEM:PROX=INS) in Komnzo; (ii) semantically similar distractors, where the model picks an English-plausible but morphosyntactically ill-formed option — e.g., great instead of gladly in Kalamang, or DEP-give=2S.IO instead of DEP-give=1S.IO in Moloko; and (iii) fine-grained form differences, such as choosing maweka over the correct moweka in Ulwa, a single-vowel contrast.

Methodology in Plain English

The researchers collected the LaTeX source of 18 open-access reference grammars from their GitHub repositories. They stripped the formatting, kept bolded keywords that mark the form under discussion, excluded chapters without usable examples (and phonetics chapters, due to inconsistent IPA formatting), and manually categorized chapters as phonology, morphology, syntax, semantics, pragmatics, or other. They then extracted each IGT example together with the "knowledge point" — the explanatory paragraph containing the grammatical rationale for that example — and recorded chapter, section, and subsection metadata. A manual cleaning pass confirmed that word counts matched gloss counts and that morphemes aligned one-to-one with glosses. They standardized on the morpheme-segmented line as input.

From this data they generated multiple-choice questions. The word carrying the bold tag became the target; the target word and its gloss were masked everywhere they appeared (though masking the translation proved unreliable, since translations paraphrase). Three distractor-generation strategies supplied three wrong options, and the correct answer was placed randomly among the four positions.

Evaluation crossed four levels of available context — S (sentence only), S+G (plus glosses), S+G+KP (plus the grammatical knowledge point), and S+G+KP+T (plus the English translation) — under a fixed prompt template that casts the model as a linguist specializing in the target language and instructs it to return only a letter. Models were instruction-tuned and run in an open-source stack (vLLM, transformers) on A6000 Ada GPUs with temperature 0.7, top-p 0.9, a 2048-token limit, repetition penalty 1.1, and sampling-based decoding. Accuracy was the metric. A separate experiment compared chain-of-thought against non-CoT prompting.

Why This Matters

Impact on research. The paper shifts evaluation away from a handful of low-resource languages and machine translation toward abstract meta-linguistic reasoning across typologically diverse languages that the models were not specifically trained on. It also provides a diagnostic account of why models fail — opaque gloss abbreviations, semantic-but-ungrammatical distractors, and minimal orthographic contrasts — which is more actionable than a single accuracy number. The finding that KPs and glosses each help independently supports the broader line of work showing LLMs can use explicit linguistic descriptions, while the contamination check cautions against over-reading high scores in this domain.

Real-world applications (as framed by the paper):

• Language documentation and revitalization. The benchmark targets 18 endangered and low-resource languages, mirroring the materials field linguists actually work with.
• Assisting morpheme segmentation and glossing. The paper motivates LLMs as tools that could suggest segmentations and glosses for undocumented data.
• Linguistic hypothesis testing. Models could be used to identify patterns or counterexamples that test grammatical hypotheses.
• Typological comparison. Structural features could be compared across many languages at once, supporting cross-linguistic research.

Industry relevance. For teams building or evaluating multilingual and low-resource LLM capabilities, LingGym offers a contamination-aware, multiple-choice benchmark with controlled context ablations. It also gives a concrete signal on prompt engineering: adding retrieved grammatical explanations and translations yields larger gains than switching to chain-of-thought or to a reasoning-tuned model, with the notable exception of DeepSeek-R1 32B.

Future Directions

1. Extend to more languages. The authors note that Grambank records reference grammars or papers for around 2,400 languages, and that their 18 languages represent only a tiny fraction of human languages.

2. Move from rule-mapping to rule-induction. Real fieldwork requires inducing rules from samples, not only applying given rules to sentences; the authors name this as a next step and cite prior work on LLM-assisted rule induction.

3. Add finer-grained syntactic categories. Because reference grammars skew heavily toward syntax (16,747 of 19,612 examples), the authors propose subdividing syntax — for example into word/constituent order, agreement, and clause structure — while acknowledging that topic coverage varies too much across languages and sources for a uniform scheme.

4. Include larger frontier models. Budget prevented evaluation of DeepSeek-R1-671B, o4, and Gemini 2.5 Pro, which the authors expect might perform more strongly.

5. Address identified error sources. The error analysis suggests incorporating fuller explanations of gloss abbreviations into prompts, which the authors flag explicitly as future work. Open questions also remain about masking free translations, since translations often leave indirect cues about the target word.

Target Audience

Field linguists and language documentation researchers interested in what current LLMs can and cannot do with descriptive materials; NLP researchers working on low-resource languages, multilingual evaluation, and benchmark design; computational linguists studying meta-linguistic knowledge in language models; and practitioners who need a controlled, contamination-aware benchmark for evaluating model reasoning over structured input rather than unstructured text.

Authors’ abstract

This paper introduces LingGym, a new benchmark that evaluates LLMs' capacity for meta-linguistic reasoning using Interlinear Glossed Text (IGT) and grammatical descriptions extracted from 18 typologically diverse reference grammars. Unlike previous work that focuses on specific downstream tasks, we assess whether LLMs can generalize linguistic inference across low-resource languages and structures not seen during training. We present a controlled evaluation task: Word-Gloss Inference, in which the model must infer a missing word and gloss from context using varying levels of linguistic information (e.g., glosses, grammatical explanations, translations). Our results show that incorporating structured linguistic cues leads to consistent improvements in reasoning performance across all models. This work highlights both the promise and current limitations of using LLMs for typologically informed linguistic analysis and low-resource language documentation.