Lemma Dilemma: On Lemma Generation Without Domain- or Language-Specific Training Data

AI summary

Overview

Research area: Natural Language Processing — contextual lemmatization and in-context learning with large language models (LLMs), with a focus on cross-lingual and out-of-domain settings.

Technical level: Intermediate. The core comparison (prompting LLMs vs. fine-tuning an encoder) is conceptually straightforward, but the paper assumes familiarity with lemmatization, sequence labeling, cross-lingual transfer, and metrics such as word and sentence accuracy.

Scope: The paper empirically compares in-context lemma generation by four large instruction-tuned LLMs against a fine-tuned XLM-RoBERTa large encoder across 12 languages, under out-of-domain and no-target-language-training-data conditions.

What This Paper Is About

Lemmatization — converting inflected words (e.g. chose, chosen) to their dictionary form (e.g. choose) — is normally solved with supervised encoder models that need large amounts of annotated data and degrade when applied out-of-domain. The authors ask whether the latest LLMs can instead generate correct lemmas directly in context, without any domain- or language-specific fine-tuning, and how that option compares to traditional supervised and cross-lingual transfer approaches. They test this across 12 languages of varying morphological complexity, including high-inflection and lower-resource ones.

Key Contributions

1. The first empirical investigation of the ability of current-generation LLMs to perform in-context lemma generation across languages of different morphological complexity.
2. A direct comparison, in out-of-domain settings, between LLMs generating lemmas in context and an encoder-only model (XLM-RoBERTa large) fine-tuned on gold data from a different distribution.
3. A comparative analysis of three strategies for data-scarce target languages — model-transfer, data-transfer (translate-train), and direct in-context lemma generation — all using English as the source language.
4. A systematic prompt-design study (basic vs. full prompts, sentence-as-string vs. sentence-as-word-list inputs, zero-shot vs. 1–5 shot, and three example-selection strategies), plus an error analysis and a model-scale analysis.

Main Findings

• Best prompt configuration: The optimal setting combined the basic prompt with 4-shot examples, the input given as a list of words, and examples selected by ranking development-set sentences by the number of errors the model produced. Adding examples improved results substantially, especially for Russian and Basque.
• Input format matters: Representing the sentence as separated tokens reduced hallucination and gave higher accuracy than passing the sentence as one string.
• LLMs beat the encoder out-of-domain for some languages: In the different-distribution experiments (Table 3), directly generating lemmas with Claude and Mistral outperformed the fine-tuned XLM-RoBERTa large encoder for Turkish, Czech, and Russian, while XLM-RoBERTa was superior for Basque and Spanish.
• Claude-3.7-Sonnet leads among LLMs: It achieved the highest accuracy for 6 out of 12 corpora in Table 3, and the highest accuracy for all languages except English in the parallel-corpus experiment (Table 4).
• Open-weights Mistral is competitive: Mistral-Large-Instruct-2407 significantly outperformed XLM-RoBERTa large in 6 of the 11 evaluation settings in Table 3.
• Cross-lingual transfer is weaker: Model-transfer (fine-tuning on English) produced an average word accuracy of 0.75 and sentence accuracy of 0.00, and data-transfer produced 0.91 word accuracy / 0.23 sentence accuracy, both below direct in-context generation with Claude (0.95 / 0.43).
• LLMs can rival in-domain fine-tuning: Direct lemma generation with Claude or Mistral was superior to even in-domain (monolingual) fine-tuning of XLM-RoBERTa large for 7 out of the 12 languages, although XLM-RoBERTa remained best for English, Swedish, Italian, French, and Turkish.
• Data-transfer quality: Data-transfer exceeded direct in-context generation with LLaMa or Qwen for 6 out of 11 languages, which the authors attribute to the high quality of Claude-produced translations.
• Model scale matters: Larger models performed much better. Average word accuracies in Table 5 were 0.74 for Qwen-2.5-7B, 0.87 for Qwen-2.5-32B, 0.90 for Qwen-2.5-72B, 0.79 for Ministral-8B, and 0.93 for Mistral-Large-Instruct-2407. LLaMa-3.1-8B failed to generate coherent output for 10 out of 12 languages.
• Recurring error types: The discussion lists randomly generated output (stray quotation marks, duplicate outputs, extra explanations), modified wordforms (changing or lowercasing input words), arbitrarily skipping words, and failures on lemmas not present in the few-shot examples (e.g. Spanish definite articles la, los, las).
• Stability: Standard deviation across three runs was consistently ≤ 0.02 for both word and sentence accuracy; Claude-3.7-Sonnet was run only once due to computational cost.
• Sentence accuracy as a metric: The authors note that word and sentence accuracy diverged meaningfully only for Turkish, Swedish, and English, with French and Italian differences visible only in sentence accuracy.

Methodology in Plain English

The authors set up three experiment groups. First, they tested four prompt designs with Mistral-Large-Instruct-2407 on English, Spanish, Russian, and Basque, varying whether the prompt was a bare task description or included explicit lemmatization instructions, whether the sentence was fed as one string or as a list of words, and whether 0 to 5 worked examples were included. Examples were chosen manually, randomly, or by picking development sentences where the model made the most errors. They judged prompts by word accuracy, sentence accuracy, and the number of missing words, wrong words, and hallucinations.

Second, they took an English-specialized scenario: XLM-RoBERTa large was fine-tuned on gold-annotated corpora for six languages (batch size 16, weight decay 0.01, learning rate 5e-5, 20 epochs) and tested on corpora from a different distribution, comparing against LLMs using the best prompt from step one.

Third, they simulated missing target-language data using parallel PUD corpora for 12 languages. They compared model-transfer (train an XLM-RoBERTa lemmatizer on English only), data-transfer (use Claude-3.7-Sonnet to translate the English PUD training set into 11 target languages and generate lemmas, then fine-tune XLM-RoBERTa on that), and direct in-context generation with the four LLMs. Evaluation used average word and sentence accuracy over three runs, with McNemar's test at α = .05 for statistical significance.

Why This Matters

Impact on research: The paper challenges the assumption that lemmatization requires task-specific supervised training. It provides the first evidence that large instruction-tuned LLMs can match or beat fine-tuned encoders out-of-domain for several languages, and it shows that cross-lingual transfer methods (model-transfer, data-transfer) are not necessarily the best answer to data scarcity for this task. It also argues for wider use of sentence-level accuracy alongside word accuracy.

Real-world applications:

• Preprocessing pipelines for information extraction, named entity recognition, and sentiment analysis, where lemmatization is a standard early step.
• Processing of morphologically rich or lower-resource languages where annotated treebanks are scarce or unavailable.
• Adapting existing NLP tools to new domains (e.g. social media, legal, biomedical text) without collecting and annotating in-domain training data.
• Rapid prototyping and deployment scenarios where a few prompting examples are cheaper than assembling a labeled corpus.

Industry relevance: The results suggest organizations can substitute prompt engineering with a large model for the costly annotation and fine-tuning cycle in some settings. The authors also report a cost side: all experiments used NVIDIA A100 80GB GPUs, totaling roughly 300 hours of processing time, 75 kWh of energy, and 32.4 kg of CO2 emissions. The finding that model scale strongly affects quality and that the best-performing model (Claude-3.7-Sonnet) was the only one evaluated with closed weights has practical implications for deployment choices.

Future Directions

• Comparing a broader spectrum of large-scale language models, since the current evaluation does not cover the full space of available architectures.
• Expanding linguistic coverage, given that evaluation datasets for low-resource and morphologically complex languages remain scarce and limit cross-linguistic validation.
• Systematic exploration of alternative prompt variations and instructional formats, which the authors did not investigate exhaustively.
• Deeper analysis of LLM failure modes for lemmatization, particularly for languages with more complex morphology, and of how example selection can be optimized (which the authors suggest may be especially beneficial for low-resource languages).

Target Audience

NLP researchers working on lemmatization, morphology, and cross-lingual transfer; practitioners building multilingual or domain-adapted text-processing pipelines; and engineers deciding between fine-tuning encoders and prompting LLMs for sequence-labeling tasks. Readers with a basic understanding of NLP tasks and evaluation metrics will get the most from it, while those interested in low-resource and morphologically complex languages will find the cross-lingual and language-selection analysis particularly relevant.

Authors’ abstract

Lemmatization is the task of transforming all words in a given text to their dictionary forms. While large language models (LLMs) have demonstrated their ability to achieve competitive results across a wide range of NLP tasks, there is no prior evidence of how effective they are in the contextual lemmatization task. In this paper, we empirically investigate the capacity of the latest generation of LLMs to perform in-context lemmatization, comparing it to the traditional fully supervised approach. In particular, we consider the setting in which supervised training data is not available for a target domain or language, comparing (i) encoder-only supervised approaches, fine-tuned out-of-domain, and (ii) cross-lingual methods, against direct in-context lemma generation with LLMs. Our experimental investigation across 12 languages of different morphological complexity finds that, while encoders remain competitive in out-of-domain settings when fine-tuned on gold data, current LLMs reach state-of-the-art results for most languages by directly generating lemmas in-context without prior fine-tuning, provided just with a few examples. Data and code available upon publication: https://github.com/oltoporkov/lemma-dilemma