Morphology and Lexical Structure

A search or extraction system encounters related strings that share meaning imperfectly.

One of those affixes is not merely a linguist's category. It is an administered international standard, and it has been rewritten twice under load. The ending -mab had been used since the early 1990s to name over five hundred monoclonal antibodies. In May 2017 the WHO cut part of it away: the species infix — -zu-, -xi-, -u-, -o- and the rest — was discontinued because it was being used as a marketing tool. In November 2021 the WHO dropped the stem -mab as well, splitting it into four suffixes: -tug for unmodified immunoglobulins, -bart for engineered or artificial ones, -mig for bi- and multi-specific ones, -ment for fragments. The AMA's USAN Council adopted the new scheme effective December 2021 and said why: “Recognizing the need for continued revisions because of crowding in the -mab stem class (over 800 monoclonal antibody names have now been selected), the USAN Council and INN Experts have revised the nomenclature scheme for monoclonal antibodies.”

Names already published are not changed retroactively, so both schemes now coexist in the literature. That is the practical lesson for anyone who ships a suffix list. An affix inventory frozen before December 2021 does not merely miss the new drugs. It misclassifies every one of them, while continuing to work on the eight hundred named before. A morphological rule is dated evidence, not a fact about the language.

Their outputs may look similar. Their guarantees differ.

Stemming applies rules that strip or replace surface endings. It is a fast heuristic, its rules are language-specific, it may produce nonwords, and it is good enough for rough retrieval. Lemmatization maps an inflected form to a dictionary lemma using linguistic analysis. It needs part-of-speech context, it returns real lexical base forms, and it can be ambiguous. Morphological analysis describes a lemma plus grammatical or derivational features — tense, number, case, gender, person. It can return several competing analyses for one string, and it is rich but resource-intensive.

The difference is not a matter of degree, and the rest of this lesson measures it. The stemmer collapses seven distinct English words into a four-letter string that is not a word at all. Morphological analysis is not an ideal either: it is a published standard, with a fixed tag inventory and a versioned release history. Which one a pipeline needs is a question about the downstream task. It has been answered experimentally more than once.

Not every language expresses these layers with separate, easily visible pieces. The surface wordform is the exact string observed in context. Beneath it sits segmentation or construction: roots, stems, affixes, clitics, reduplication, or compounds. Then the lemma and lexical category. Then the morphosyntactic features — number, case, tense, aspect, mood, person, gender, or definiteness. Last, the contextual interpretation that surrounding syntax and meaning select.

Three of those layers are not a teaching device. They are a standard with version numbers. Universal Dependencies fixes the morphological layer at exactly three levels: a lemma, a part-of-speech tag drawn from a fixed list of 17 universal tags, and a set of universal morphological features. Release v2.5, of 15 November 2019, contains 157 treebanks representing 90 languages across 20 language families. The sizes are not comparable. Skolt Sámi has 36 sentences and 321 words. German has more than 200,000 sentences and nearly 4 million words. The people who built the collection state its limit themselves: “The selection is very heavily biased towards Indo-European languages (48 out of 90), and towards a few branches of this family”. A layer diagram that looks universal is, in the data you can actually download, mostly Indo-European.

The layers have been catalogued a second way. UniMorph publishes “broad-coverage instantiated normalized morphological inflection tables for hundreds of diverse world languages” under “a language-independent feature schema for rich morphological annotation”. Version 4.0, in 2022, amended that schema “to use a hierarchical structure that is needed for morphological phenomena like multiple-argument agreement and case stacking”, and “augmented the database with morpheme segmentation for 16 languages”. A feature schema that has to grow a hierarchy in order to record agreement is telling you something about the layer above the wordform. So is the number 16. That is how much of the second layer is annotated anywhere, by the project whose whole business is annotating it.

An isolating language may express grammatical relationships through word order and separate particles, while an agglutinative language can attach many relatively regular morphemes. Fusional systems may encode several features in one ending. Polysynthetic forms can express what other languages write as a clause.

The intuition that follows — richer morphology means a harder language to model — was measured, published, and then failed to replicate. In 2018 Cotterell and Mielke studied 21 Europarl languages and found “complex inflectional morphology to be a cause of performance differences among languages”. The next year the two of them, with three colleagues, extended the study to 69 languages from 13 language families over 106 Bible translations. It did not reproduce. Morphological counting complexity showed a Spearman's ρ of only .091 for the BPE model and .110 for the character model, with p > .6. WALS feature 26A, “Prefixing vs. Suffixing in Inflectional Morphology”, showed no significant difference between categories, at p > .3 and p > .7 for the two models. Their own figure caption says it: “MCC does not predict difficulty on Europarl”.

The non-replication went in the abstract rather than a footnote: “Trying to answer the question of what features difficult languages have in common, we try and fail to reproduce our earlier (Cotterell et al., 2018) observation about morphological complexity and instead reveal far simpler statistics of the data that seem to drive complexity in a much larger sample.” What did predict difficulty was not typology at all: “Our most predictive feature, however, is the size of the word inventory.” That association reached p < 1e-11, and only for the BPE model.

So the typological labels describe tendencies, not rigid boxes. Even the leading quantitative version of the claim did not survive a larger sample. Inspect the target languages. Do not assume that spaces, suffix lists, or English part-of-speech categories transfer unchanged, and do not assume a typological label tells you how hard the modeling will be.

Use controlled comparisons rather than assuming linguistic analysis always improves a learned model. First, identify a concrete failure: sparsity, inflection mismatch, compound variation, or feature-dependent errors. Second, choose the representation to test — lemma, stem, morpheme segments, morphological features, or character patterns. Third, preserve the surface form, keeping original text and alignment for names, evidence, and reconstruction. Fourth, evaluate by language and feature, measuring gains and regressions for tense, case, number, compounds, and rare forms. Fifth, keep a simpler baseline, because modern subword or character models may already capture enough structure.

Steps two and five are the ones with published evidence behind them, and it points both ways. After the 2019 non-replication the question was re-opened with more data. In March 2021 Park and Zhang, with four colleagues, compiled 145 Bible translations in 92 languages and 11 more WALS morphology features than the earlier study had used. Their conclusion: “BPE model surprisal is associated with 6 out of the 12 studied WALS morphology features, indicating that there are aspects of some languages’ morphology that BPE does not help mitigate.” Their abstract puts the same result in modeling terms: “We find that several morphological measures are significantly associated with higher surprisal when LSTM models are trained with BPE-segmented data.” They confirmed the earlier negative result on its own terms. Feature 26A, the single feature Mielke and colleagues had tested, indeed showed no correlation. Six of twelve features did.

That is what makes step two a measurement rather than advice. Park and colleagues also found that morphologically motivated segmentation — Morfessor, and FST-augmented BPE where available — both models the data better and weakens the association between a language's morphology and its modeling difficulty. So the simpler baseline in step five is a trade-off with a published head-to-head behind it, not a default to be assumed either way. Run the comparison on your languages and your task, and report which of the two the numbers favour.

Select twenty queries whose relevant documents use different inflections, compounds, or derivations. Compare word matching, stemming, lemmatization, subword retrieval, and character n-grams. Record improved matches and new false positives. A successful method should recover intended variation without merging opposites, names, or unrelated words.

This experiment has a published version with numbers you can measure yourself against, and it was run in a language where the answer differs from the English one. In 2004 Braschler and Ripplinger tested a spectrum of stemming and decompounding methods on a unified German collection: Frankfurter Rundschau and Der Spiegel articles from the CLEF 2000 and 2001 data, with the CLEF 2001 SDA newswire removed, and 85 of the 90 topics from the two years. Their headline: “Compared to a system without stemming, we obtained performance gains measured in mean average precision of up to 23% for short (T) and up to 11% for long queries (TDN).” Recall rose 12% and 4% respectively. Decompounding contributed more than stemming did — 16% to 34% for short queries and 9% to 28% for long ones — and the best decompounding runs came in 33% to 60% above the no-stemming baseline. The published abstract states the finding as a rule of thumb: “The main findings are that stemming is beneficial even when using a simple approach, and that carefully designed decompounding, the splitting of compound words, remarkably boosts performance.”

So the same operation that bought English retrieval a non-significant change bought German up to 23% in mean average precision, and splitting compounds bought more than that again. The IR textbook reports the pattern independently: “The situation is different for languages with much more morphology (such as Spanish, German, and Finnish). Results in the European CLEF evaluations have repeatedly shown quite large gains from the use of stemmers (and compound splitting for languages like German)”. Braschler and Ripplinger also record that the English literature never fully settled: “Where Harman (1991) reported that stemming gives no benefit, Frakes (1992) and Hull (1996) claim at least a small benefit.” Run your twenty queries, then ask which of those two worlds your language and collection resemble. The answer is a measurement, not a preference.