Confounding Factors in Relating Model Performance to Morphology

AI summary

Overview

• Research area: Natural Language Processing — specifically the methodology of comparing tokenizers and language models across languages with different morphological systems.
• Technical level: Advanced (assumes familiarity with subword tokenization algorithms such as BPE and ULM, perplexity, and morphological typology).
• Scope: The paper catalogues confounding factors in experiments linking morphology to language-model performance, re-assesses three hypotheses from Arnett and Bergen (2025) about agglutinative-versus-fusional perplexity gaps, and proposes token bigram metrics (accessor variety and entropic efficiency) as intrinsic, annotation-free proxies for morphological complexity.

What This Paper Is About

Research on whether morphology makes some languages harder to model has produced contradictory results — some studies find agglutinative languages are harder, others find no difference. This paper argues the contradiction comes from experimental confounds rather than from language properties, and demonstrates concretely how those confounds operate in a specific recent study. It then proposes a new family of metrics based on neighbouring token pairs that both predicts causal language modeling difficulty and tracks morphological complexity as a gradient rather than a binary category.

Key Contributions

1. A list of confounding factors that any experiment relating morphology to language modeling must control for, framed as criteria for an "ideal" experiment. The six factors named are: language set, grouping, tokenization algorithm, vocabulary size versus data size, corpus domain, and performance indicator.
2. A re-assessment of the three hypotheses proposed by Arnett and Bergen (2025) for the perplexity gap between agglutinative languages (ALs) and fusional languages (FLs): H1 morphological alignment of tokenization, H2 tokenization efficiency, and H3 dataset size. For each, the authors identify confounds and suggest methodological improvements.
3. Token bigram metrics — accessor variety (AV), accessor uniqueness (AU), and entropic efficiency (η) — as an intrinsic way to predict the difficulty of causal language modeling, computed on the same units a language model uses.
4. The finding that these bigram metrics are gradient proxies for morphological complexity that require no expert annotation, offering a new hypothesis for why causal language models may struggle more with agglutination.

Main Findings

• Conflicting literature is a methodology problem, not a language problem. Previous work has variously concluded ALs are harder to model (Cotterell et al., 2018; Gerz et al., 2018a, 2018b) or that there is no difference (Mielke et al., 2019; Arnett and Bergen, 2025). The paper attributes this to uncontrolled variables rather than to morphology itself.

• MorphScore measures stem-suffix recall only, not full alignment. MorphScore's datasets contain one morpheme boundary per word — the stem-suffix boundary — and the metric tracks micro-averaged recall in two modes. The paper argues stem-suffix boundaries are not explanatory for ALs, because losing one boundary in an AL can hurt as much as losing several in an FL. Missing agglutinative suffix-suffix boundaries can strip a morph of half its length and misses can cascade across multiple tokens.

• Grouping by typology hides large individual variation. Arnett and Bergen find AL tokenization is more aligned than FL tokenization based on averages across MorphScore's 22 languages, evenly divided between ALs and FLs. But English (an FL) has the second-highest MorphScore of all languages, and five FLs score higher than Turkish (an AL).

• Full-alignment results do not separate the groups. Using MorphyNet segmentations (13 languages, 100k to over 1 million examples each) and MorphoChallenge (about 1000 examples), the paper finds suffix-suffix boundaries are missed much more than stem-suffix boundaries within a language in almost all cases, at an unpredictable rate, and that comparing scores across languages shows neither grouping consistently having a higher F1.

• Token unigram efficiency metrics do not explain the gap. Arnett and Bergen computed corpus token count (CTC) and Rényi efficiency (RE) on the multi-parallel FLORES-200 dataset and found higher RE for AL tokenizers, which they call undesirable. The paper agrees with the conclusion that unigram metrics are poor explanations, and notes that in larger corpora there are minimal RE differences between an FL such as Romanian and an AL such as Finnish. It also points out that calling higher entropy undesirable conflicts with Zouhar et al. (2023).

• The dataset-size conclusion is confounded by simultaneous changes. Arnett and Bergen observed the AL/FL perplexity gap shrink to a p-value of 7.7% when using Goldfish models whose data were scaled by byte-premium (BP), and concluded BP explains away morphology. The paper shows that the earlier and later experiments differ in vocabulary sizes, training data, model sizes, and languages studied (see the paper's Table 3), so the effect of BP cannot be isolated. Per hypothesis, the paper reports the number of languages actually analysed: H1 has 22 language-script pairs (11 ALs, 11 FLs, 32k vocabulary, 10k lines of tokenizer data); H2 reports 63 but the actual number is 53 (37 ALs, 16 FLs, again 32k vocabulary and 10k lines); H3 reports 154 but the actual number is 149 (85 ALs, 64 FLs, 50k vocabulary, 100 MiB tokenizer data).

• The three experiments barely share languages. Only 3 languages appear in all three hypotheses (H1 ∩ H2 ∩ H3 = 3), 3 in H1 ∩ H2, 22 in H1 ∩ H3, and 52 in H2 ∩ H3, with a union of 145. Conclusions are therefore drawn predominantly about different languages. H2 has 52 and H3 145 unique languages; all duplicate language-script combinations are ALs.

• Comparing perplexities across languages is not straightforward. The paper argues that comparing PPLs of different segmentations of the same test text is valid only after rescaling to a shared unit such as characters, and that comparing models targeting different test sets in different languages compares both different segmentations and different underlying distributions. It gives a hypothetical example (Table 4) where a Spanish sentence has two valid parallel English translations with PPLs of 22 and 18, so which model (PPL 20 or 18) looks "better" depends on an arbitrary choice of translation. Perplexity also ranges from 1 to infinity, producing outliers, and the paper states that outliers exist in Arnett and Bergen's results, showing byte-premiums are not a complete solution.

• Token bigram metrics recover coarse typology and reveal finer structure. On a multi-parallel aligned subset of EuroParl, AV recovers the coarse groupings with ALs having the highest AV, and η follows the same trend. Within FLs, higher AV values point to compounding languages such as German and Danish, while English and Romanian sit lower. Word-based metrics recover groupings only somewhat; token unigram metrics look rather even across EuroParl languages.

• Domain matters. Comparing EuroParl (21 languages) and FineWeb (63 languages), with 19 in the intersection and 65 in the union, AV and η are always lower for EuroParl than FineWeb, while MATTR moves in the opposite direction.

• Writing systems are part of the picture. Without pretokenization, the metrics place ALs near the top but also favour languages using long words written in systems with large character inventories, such as abugidas. The paper notes that since morphology is encoded in writing systems, byte-premiums may be a consequence of morphology rather than an alternative explanation for it.

Methodology in Plain English

The authors first abstract away from any single experiment. They read the existing literature on morphology and language modeling, identify every variable that changes between studies, and write those out as a checklist of confounds — for example, whether the languages compared are the same across sub-analyses, whether the tokenizer is the same, whether the vocabulary size is appropriate for the amount of training data, and whether the evaluation metric is comparable across languages. This checklist effectively defines what an ideal experiment would look like, from which researchers can work backwards to something feasible.

They then take one prominent recent study (Arnett and Bergen, 2025) and go through its three hypotheses one at a time, checking each against the checklist and re-running parts of the analysis with different data where possible. For the alignment hypothesis, they replace the study's non-public tokenizers with openly available Goldfish ULM tokenizers (50k vocabulary, 1 GiB of data, versus the original 32k vocabulary and 10k lines), and build two additional diagnostic datasets from words with at least three morphemes: one testing only the stem-suffix boundary and one testing only the suffix-suffix boundaries. For the efficiency and data-size hypotheses they point to variables that changed simultaneously and propose paired tests to isolate them.

Finally, they design new metrics. Borrowing the idea of accessor variety from Harris (1955) and Feng et al. (2004), they count how many distinct token types can precede or follow a given token type in a corpus, compute their uniqueness relative to total occurrences, and measure how uniformly those neighbours are distributed using Shannon efficiency. All are computed in fixed-size windows of 1000 accessors, mimicking MATTR (Covington and McFall, 2010). The intuition is that a language model faces fewer choices when a token has few, predictable follow-ups, and more choices when it has many, equally likely ones. They compute these metrics on EuroParl and on FineWeb 1 & 2, selecting languages that have a 1 GiB Goldfish tokenizer and at least 200k lines of data so that data amounts are comparable.

Why This Matters

Impact on research. The paper reframes a live disagreement as a measurement artefact and supplies a reusable checklist, which should make cross-study comparisons in multilingual NLP more trustworthy. It also introduces a metric family that is intrinsic (no trained model or human annotation required) and gradient (no binary typological labels), addressing a known weakness of coarse groupings.

Real-world applications:

• Tokenizer and vocabulary design for multilingual models — understanding which languages impose more next-token ambiguity per vocabulary budget informs the vocabulary-size-versus-data-size trade-off the paper flags as a confound.
• Fair benchmarking across languages — knowing when cross-lingual perplexity comparisons are valid (same tokenizer and test set, or rescaled to a shared unit) affects how leaderboards and evaluation suites are built.
• Low-resource language prioritization — byte-premium-style data scaling decisions (allocating e.g. 30 MiB instead of 10 MiB to a language needing three times the bytes) rest on whether morphology or encoding explains the gap.
• Corpus quality control — the corpus-domain finding (EuroParl versus FineWeb moving metrics in opposite or consistent directions) matters for anyone assuming a single corpus represents a language.

Industry relevance. Companies training multilingual models must decide how much data per language, what vocabulary size, and which tokenizer, all under cost constraints. This paper shows those choices are entangled with how languages are written and structured, and that some commonly cited justifications for those choices rest on confounded evidence.

Future Directions

• Build the ideal experiment. The paper states that an experiment with parallel training and testing data for a large set of languages, relating multiple intrinsic metrics (alignment and efficiency) to comparable language-model metrics, is a large-scale effort outside its scope.
• Obtain full reference segmentations for more languages. The authors note that high-quality segmentations are rare and their coverage is limited, which prevented them from drawing broad conclusions in the full-alignment analysis.
• Isolate byte-premium effects. A paired t-test between models sharing language, tokenizer, architecture, and test set, differing only in whether training used 1× 10 MiB or BPL × 10 MiB, would be needed to properly test the data-size hypothesis.
• Develop comparable and informative intrinsic language-modeling metrics. The paper explicitly argues for future work here, since perplexity across languages is hard to compare and contains outliers.
• Test the causal claim. The proposed hypothesis that higher AV and η are causally linked to higher perplexity — and thereby explain the AL/FL gap — is offered as a hypothesis requiring further validation.

Target Audience

Researchers and practitioners working on multilingual NLP, subword tokenization, and language model evaluation who need to design or interpret cross-linguistic comparisons. It is also relevant to computational morphologists and typologists interested in how morphological structure shows up in model behaviour, and to engineers choosing tokenizers, vocabulary sizes, and per-language data budgets for multilingual systems. Readers without background in tokenization algorithms and perplexity will find the paper demanding, since it assumes familiarity with these concepts throughout.

Authors’ abstract

The extent to which individual language characteristics influence tokenization and language modeling is an open question. Differences in morphological systems have been suggested as both unimportant and crucial to consider (Cotterell et al., 2018; Gerz et al., 2018a; Park et al., 2021, inter alia). We argue this conflicting evidence is due to confounding factors in experimental setups, making it hard to compare results and draw conclusions. We identify confounding factors in analyses trying to answer the question of whether, and how, morphology relates to language modeling. Next, we re-assess three hypotheses by Arnett & Bergen (2025) for why modeling agglutinative languages results in higher perplexities than fusional languages: they look at morphological alignment of tokenization, tokenization efficiency, and dataset size. We show that each conclusion includes confounding factors. Finally, we introduce token bigram metrics as an intrinsic way to predict the difficulty of causal language modeling, and find that they are gradient proxies for morphological complexity that do not require expert annotation. Ultimately, we outline necessities to reliably answer whether, and how, morphology relates to language modeling.