Creating ConLangs to Probe the Metalinguistic Grammatical Knowledge of LLMs

AI summary

Overview

Research area: Natural Language Processing / computational linguistics; specifically the metalinguistic grammatical knowledge of large language models, probed through the construction of artificial "constructed languages" (ConLangs).

Technical level: Intermediate. The paper assumes familiarity with basic linguistic terminology (morphology, syntax, case marking, glossing) and with standard LLM prompting setups, though each concept is defined.

Scope: The paper presents IASC, a modular agentic LLM system for building ConLangs, and uses its morphosyntax module as a benchmark to measure how well six LLMs apply abstract grammatical specifications to sentences.

What This Paper Is About

Large language models can discuss languages, but it is unclear whether they can actually manipulate language using abstract grammatical concepts, as a trained linguist would. The authors build a system that generates constructed languages from user-specified linguistic parameters, then turn its morphosyntax stage into a test: given an English sentence, a target grammar, and an instruction to change, say, word order or case marking, can the model produce the correct output? The goal is to measure metalinguistic grammatical knowledge separately from encyclopedic knowledge or translation ability in any one language.

Key Contributions

1. IASC (Interactive Agentic System for ConLangs) — a modular system that generates ConLangs using module-specific prompt sets for phonology/phonotactics, morphosyntax, lexicon, orthography, and grammatical handbook writing, plus translation of further texts into the language. IASC is pronounced /ˈiː.ask/, and iasc means 'fish' in Modern Irish.
2. An agentic refinement loop — several modules can improve their output based on automatically generated commentary on a previous step; the authors also describe a novel agentic approach to generating phonological rules from attested sets of sound changes in the Appendix.
3. A new evaluation dataset for metalinguistic morphosyntax — 45 source sentences crossed with nine ConLang feature sets, amounting to 405 sentences, with gold glosses created by one of the authors, a trained linguist. Released under CC BY-SA 4.0 at huggingface.co/datasets/ctaguchi/conlang_eval_dataset.
4. A set of morphosyntax-specific metrics and an empirical comparison of six LLMs — TER, SER, MFER, MSER, and lemma F1, applied across nine typologically diverse feature sets.

Main Findings

• Typologically unusual targets are harder. All models struggled most with the 'hard' language, which combines atypical head-initial/head-final mixing with complex and unusual morphology. It had the highest average MSER at 85.21, while 'welsh' was easiest at 28.26.
• Uncommon word orders produce higher syntactic errors. Higher SERs appeared for 'fijian' (VOS), 'mizo' (OSV), and 'hard' (OSV), while 'french' (SVO), 'arabic' (VSO), and 'welsh' (VSO) yielded lower errors. Classifying SOV, SVO, and VSO as common and the rest as uncommon yields a weak significant correlation between word order and SER (r_pb = 0.29, p = 0.0026 < 0.05).
• Some models achieved perfect syntax on VSO languages. Claude 3.5 Sonnet, Gemini 2.5 Pro, and GPT-5-mini yielded perfect SER scores (0.00) on 'arabic' and 'welsh'.
• Lemmatization is harder for analytic languages. Treating 'fijian', 'mizo', and 'vietnamese' as analytic and the others as synthetic (following Bickel and Nichols 2013), there was a weak significant correlation between inflectional degree and lemma F1 (r_pb = 0.32, p = 0.00082 < 0.05). Outputs for analytic languages sometimes left inflected nouns and verbs inflected.
• Few-shot in-context learning helps the stronger models. With an added review stage providing a grammar summary and four example sentences with labeled glosses, Claude 3.5 Sonnet, Gemini 2.5 Flash, Gemini 2.5 Pro, and GPT-5 improved on most metrics. For example, Gemini 2.5 Pro went from MSER 54.18 to 14.68 and from SER 60.80 to 12.64.
• Smaller models did not benefit. GPT-4o-mini got slightly worse with ICL (MSER 69.16 to 68.14; TER 87.58 to 88.83), and GPT-5-mini barely changed (MSER 55.25 to 54.40).
• Bigger models separate from smaller ones. Grouping Claude 3.5 Sonnet, Gemini 2.5 Pro, and GPT-5 as larger models showed a significant difference versus the others (r_pb = −0.31, p = 0.001 < 0.05).
• Gemini 2.5 Pro was strongest overall. With few-shot ICL it outperformed the other models in all metrics except lemma F1 by a significant margin (TER 40.94, SER 12.64, MFER 16.71, MSER 14.68, LemF1 89.30).
• Qualitative example on the 'hard' language. On the sentence "The children played in the park," Gemini 2.5 Pro's ICL output came closest to the gold, following OSV order, ergative-absolutive alignment, and most morphology; GPT-4o-mini ignored morphology entirely even with few-shot examples; GPT-5 without in-context examples produced no output at all.

Methodology in Plain English

The system is built in stages. Given high-level specifications from a user, IASC first generates a Python program that produces morphemes approximating the target language's phonemes and their distribution, iterating on its own prior output. It then develops morphosyntax, generates target-language morphemes using the phonotactic grammar, builds an orthography in the desired script, writes a grammatical handbook, and invents a language name.

For the experiments, the authors focus on morphosyntax. Rather than asking the model to transform a sentence in one pass, which failed in preliminary tests because prompts became too long and complicated, they use a cumulative approach: the model transforms its own previous output at each step, with each step targeting one grammatical feature based on the grammar specification and a feature-specific prompt. The output is a gloss rather than a final translation, and the English source and prompts are in English.

Evaluation covers nine feature sets: eight named after existing languages for convenience (Arabic, Fijian, French, Hixkaryana, Mizo, Turkish, Vietnamese, Welsh) — the authors caution these names do not strictly reflect all morphosyntax details of the named languages — plus a 'hard' language with an extremely rare typological combination. Gold glosses were written by a trained linguist among the authors.

Model outputs were normalized into JSON via a lightweight model, GPT-4.1 mini, before scoring. Metrics: TER (edits divided by reference length, where shifting a word group counts as one edit), SER (TER computed on stems only, ignoring morphological feature tags, to isolate syntax), MFER (morpheme-level edit distance per reference word, ignoring word order and ignoring order within affix sequences), MSER (weighted average of SER and MFER with α = 0.5 by default), and macro lemma F1 (harmonic mean of lemma precision and recall, averaged per prediction). Six models were tested at temperature 0 where specifiable, with thinking enabled for reasoning models; each result comes from a single run.

Why This Matters

Impact on research. The paper offers a way to probe metalinguistic grammatical knowledge that is disentangled from encyclopedic knowledge about particular languages, and it does so with fine-grained, feature-by-feature control that a one-pass translation test would not allow. It also adds to the debate over whether LLMs capture abstract linguistic structure, reporting that performance tracks how typologically common a pattern is. The dataset and code are released, making the benchmark reproducible.

Real-world applications:

• Assisting conlangers, fiction writers, and game or film worldbuilders in creating consistent invented languages.
• Generating grammatical descriptions and glossed corpora that could be adapted for low-resource language documentation, which the authors discuss in the Appendix.
• Testing and stress-testing grammar engines, morphology tools, and language-learning systems against rare or unusual typological configurations.
• Serving as a diagnostic for multilingual NLP pipelines by isolating where morphology versus word order breaks down.

Industry relevance. The results give a concrete signal about which off-the-shelf models can follow complex, multi-step, rule-governed instructions: larger models (Claude 3.5 Sonnet, Gemini 2.5 Pro, GPT-5) separated significantly from smaller ones, and small models did not improve with in-context examples. That is directly useful for anyone deciding whether a smaller or cheaper model is adequate for a structured transformation task.

Future Directions

• Reducing source-language residue. Because translation is done in successive steps, the target language retains English-specific syntax such as do-support, auxiliary be in passives, and direct translations of English expressions. The authors suggest explicitly instructing the model to avoid direct translation of items like auxiliary verbs and to be creative.
• Expanding and fixing the feature inventory. Diminutives and augmentatives are not covered. Gender remains problematic: besides natural gender, it is unclear how the system should assign gender to a target word, and only masculine, feminine, and neuter are supported rather than systems like Bantu-like gender.
• Broadening morphological strategies. Morphology is limited to prefixation and suffixation; once an affix expresses a feature it is reused in all cases; feature sequences are expressed as affix sequences. The system is biased toward a strictly affixal, single-exponent, agglutinative strategy with no paradigmatic variation.
• Enlarging the evaluation set. The authors note their evaluation set, though intended to be comprehensive, could be expanded to cover a wider range of examples testing system abilities, and that model capabilities are continually changing so performance will likely improve.

Target Audience

This paper is most useful to computational linguists and NLP researchers working on the linguistic competence of LLMs or on evaluation benchmarks; to researchers interested in conlanging, language documentation, and low-resource language tooling; and to practitioners who need to know how reliably current LLMs follow complex, compositional grammatical instructions. Readers without a linguistics background will need to pick up terms like ergative-absolutive alignment, glossing, and head-marking from context, but the paper defines its key concepts as it goes.

Authors’ abstract

We present a system that uses LLMs as a tool in the development of Constructed Languages -- ConLangs, which we call IASC (Interactive Agentic System for ConLangs). The system is modular in that it creates each of the components -- phonology, morphology and syntax, lexicon, orthography, and grammatical handbook, using module-specific sets of prompts. The approach is agentic in that various modules allow for refining the output given automatically-generated commentary on a previous step. Our main goals are twofold. First, we aim to provide tools that facilitate an engaging and enjoyable experience in creating artificially constructed languages. Second, the focus of this paper is on using our ConLang framework as a novel way to explore what LLMs 'know' about language -- not what they know about any particular language or encyclopedic facts, but how much they know about and understand language and linguistic concepts. In the experiments, we particularly focus on the morphosyntax module and show that there is a fairly wide gulf in capabilities both among different LLMs and among different linguistic specifications, with it being notably easier for systems to deal with more typologically common patterns than rarer ones. All code is released.