Massively Multilingual Joint Segmentation and Glossing

AI summary

Overview

Research area: Natural Language Processing — multilingual morphological segmentation and interlinear glossing for low-resource and endangered languages.

Technical level: Intermediate. The paper assumes familiarity with sequence-to-sequence modeling, subword vs. byte-level tokenization, and low-rank adaptation, but the task itself (interlinear glossed text) is explained from first principles.

Scope: The paper introduces PolyGloss, a family of multilingual ByT5-based seq2seq models that predict morphological segmentation, interlinear glosses, and the alignment between them in a single pass, trained on an expanded version of the GlossLM corpus.

What This Paper Is About

Automated interlinear glossing models like GlossLM score well on benchmarks, but linguists found them hard to trust because they assign morpheme-level glosses to whole words without ever predicting where the morpheme boundaries actually are. This paper builds the first neural models that jointly predict morphological segmentation and glosses from raw transcribed text, so that each gloss is explicitly tied to a specific morpheme. The goal is to produce predictions that are accurate, interpretable, and adaptable to the glossing conventions a particular annotator prefers.

Key Contributions

1. An expanded and cleaned GlossLM corpus. The authors add a large set of new interlinear glossed text (IGT) examples, including 80,461 IGT instances for 37 languages from the Fieldwork dataset and an update of IMTVault to version 1.2 adding 39,741 examples; after removing 20,116 duplicates, this yields 91,416 new unique examples. The release also fixes punctuation handling, formatting errors (such as 4,882 Arapaho instances where ",." appeared inside glosses), and misaligned segmentation-gloss pairs.

2. PolyGloss, a family of multilingual joint segmentation-and-glossing models. Built by continued pretraining on ByT5 (the byt5-base checkpoint with 580M parameters) using three different task formats: multitask, concatenated, and interleaved.

3. A novel alignment score. A reference-free metric that abstracts segmentation and gloss predictions into structural "x" sequences, computes character-level edit distance normalized by the longer sequence, and converts it to a score in [0,1] where 1 is perfect.

4. Practical adaptation and evaluation tooling. Demonstrations that per-language perplexity predicts glossing accuracy (r² = 0.951), that LoRA adapters can adapt PolyGloss to a new language in minutes, and that the alignment score works as a reward function for GRPO reinforcement learning.

Main Findings

• PolyGloss sets a new state of the art on glossing. Using morpheme error rate (MER, lower is better) averaged over nine languages, PolyGloss with the interleaved format reaches 0.234, compared to 0.265 for the multitask version, 0.639 for GlossLM, 0.839 for Qwen 3 0.6B in-context learning, 0.641 for Aya Expanse 8B, and 0.559 for Gemma 3 4B. GlossLM's average is depressed because only Arapaho, Tsez, and Gitksan were explicitly in its pretraining corpus.

• Segmentation performance is far above the LLM baselines. Averaged morpheme F1 for segmentation is 0.862 for PolyGloss (interleaved), 0.860 for PolyGloss (multitask), versus 0.167 for Qwen 3 0.6B (ICL), 0.421 for Gemma 3 4B, and 0.371 for Aya Expanse 8B.

• The interleaved format produces perfect alignment. On the alignment score, PolyGloss with interleaved prediction achieves 1.000 on average across all nine languages, compared to 0.973 for the multitask model, 0.984 for Gemma 3 4B, 0.961 for Aya Expanse 8B, and 0.661 for Qwen 0.6B.

• Joint training beats separate training. In monolingual ablations, training segmentation and glossing jointly was best on all three metrics; training separate models produced accurate but misaligned outputs; a two-model pipeline suffered from error propagation; and the hard-attention approach of Girrbach (2023a) performed poorly on segmentation.

• Multilingual pretraining transfers. PolyGloss outperformed monolingual joint models on average for glossing and segmentation, though the standard error is large. For the three languages with the most training data (Arapaho, Uspanteko, and Ainu), monolingual models were slightly superior at glossing; for all other languages the multilingual model was far superior.

• Perplexity predicts glossing quality. Per-language validation perplexity correlates strongly with morpheme error rate (r² = 0.951), giving a practical way to decide whether to trust the model on an unseen language, fall back to a simpler method such as predicting the highest-frequency gloss, or show no prediction at all.

• The model adapts quickly to new languages. On Vamale, which never appears in pretraining, LoRA adapters (rank 8, 25 epochs, batch size 32, increments of 50 examples up to the full 380) reduced morpheme error rate steadily, with the largest training run taking 12 minutes on an A100. A ByT5 base model did not improve under the same setup.

• The alignment score works as a reinforcement learning reward. A GRPO pilot on Gitksan (β = 0.1, learning rate 5E-5, batch and group sizes of 8, 50 epochs, temperature 0.6, top-p 0.9, repetition penalty 1.05) improved all three metrics slightly.

• Longer outputs are not a problem. Filtering the test set to inputs in the 75th percentile of length per language, glossing MER for the interleaved model was 0.226 versus 0.234 on the full test set, indicating the doubled output length of the interleaved and concatenated formats does not degrade accuracy.

• Decoder-only LLMs did not converge well. Continuing training on Qwen 3 0.6B produced much worse scores (average MER 0.693, segmentation F1 0.555, alignment 0.997), which the authors attribute to subword tokenization, extensive prior instruction tuning and reinforcement learning, and high-variance gradients given the model size relative to dataset size.

Methodology in Plain English

The authors frame the task as: given a transcribed sentence (and optionally an English translation), produce the morphological segmentation and the gloss line.

1. Data. They start from the GlossLM corpus, standardize punctuation and formatting across sources, add new IGT data, remove duplicates, and mark or blank out pairs where segmentation and glosses do not match in word or morpheme count. Misaligned examples are kept only in the training split so evaluation is not affected.

2. Three output formats. In multitask, segmentation and glossing are trained as separate examples. In concatenated, the model generates the segmentation first and the glosses second, letting it condition glosses on segments. In interleaved, each gloss label is immediately followed by its morpheme in parentheses, which enforces alignment structurally.

3. Model and training. They continue pretraining ByT5, a byte-level encoder-decoder chosen because byte tokenization avoids problems for rare languages that subword tokenizers handle poorly. Training used bf16, AdamW, a linear warmup over the first 3% of steps, cosine decay, and gradient clipping with max norm 1 on 4 GH200 GPUs, with a learning rate of 5E-5, batch size 64, and 15 epochs. Evaluation uses beam search with 2 beams.

4. Evaluation. Glossing is measured primarily with morpheme error rate, computed by inserting [SEP] tokens between per-word glosses and taking normalized edit distance; the paper also reports word and character error rates, BLEU, and morpheme and word accuracy. Segmentation uses the modified F1 of Mager et al. (2020), plus character edit distance and whole-word accuracy. A new alignment score measures whether segmentation and glosses line up with each other, independent of correctness.

5. Comparisons. Multilingual comparisons include the pretrained GlossLM model and in-context learning with Qwen 3 0.6B (with thinking), Cohere Aya Expanse 8B, and Google Gemma 3 4B, using ten chrF-retrieved examples. Monolingual comparisons include separately finetuned ByT5 models, a jointly finetuned ByT5 model, a two-model pipeline, and a hard attention transformer.

Why This Matters

For research, this work reframes glossing as a joint segmentation-and-glossing problem and argues that alignment between the two output lines is a first-class evaluation target, not an afterthought. The alignment score is reference-free, so it can also serve as an RL reward or a data-quality filter without gold labels. The release of an expanded, standardized corpus and open weights also lowers the bar for future multilingual work on endangered languages.

Real-world applications:

• Language documentation software. Integrating PolyGloss into tools like ELAN or FLEx to draft segmentation and gloss lines for transcribed recordings, which annotators then accept, reject, or edit.
• Confidence-based triage. Using per-language perplexity to decide whether to show model predictions, fall back to a frequency-based lookup, or show nothing, avoiding the low-quality outputs linguists reported in earlier user studies.
• Convention adaptation. A linguist who prefers a different glossing schema can train a small LoRA adapter overnight so the model follows their conventions.
• Downstream language technology. IGT is used to build taggers, searchable text databases, educational tools, and machine translation systems, all of which benefit from faster and more consistent annotation.

Industry relevance: the paper targets the annotation-tooling and language-technology space, arguing that a single serveable multilingual model is preferable to linguists training and hosting their own per-language models, and it explicitly raises data sovereignty concerns as a reason to prefer open-weight models that can be finetuned locally over closed-source APIs.

Future Directions

• Integration into documentation workflows. The authors plan to work with the developers of ELAN and FLEx to embed the model, predicting segmentation and gloss lines and filtering suggestions by model confidence.
• Scaling reinforcement learning. The GRPO pilot on Gitksan improved all metrics slightly; the authors suggest scaling this to the full training dataset as a post-training step, and using it for languages that lack gold segmentation labels.
• Better decoder-only models. Qwen-based PolyGloss underperformed badly, but the authors expect the right hyperparameters could bring instruction-tuned decoder LLMs to at least ByT5-level accuracy.
• Broader language coverage. Languages absent from or underrepresented in the pretraining corpus remain a problem; LoRA adaptation is offered as the current answer, but the corpus itself could still be expanded.

Target Audience

Computational linguists and NLP researchers working on low-resource languages, morphological segmentation, or interlinear glossing; documentary linguists and language revitalization practitioners who might use or evaluate such tools; and developers building annotation software and linguistic data pipelines. Readers interested in evaluation design will also find the alignment score and the morpheme-error-rate discussion useful, while those focused on model adaptation will benefit from the LoRA and GRPO experiments.

Authors’ abstract

Automated interlinear gloss prediction with neural networks is a promising approach to accelerate language documentation efforts. However, while state-of-the-art models like GlossLM achieve high scores on glossing benchmarks, user studies with linguists have found critical barriers to the usefulness of such models in real-world scenarios. In particular, existing models typically generate morpheme-level glosses but assign them to whole words without predicting the actual morpheme boundaries, making the predictions less interpretable and thus untrustworthy to human annotators. We conduct the first study on neural models that jointly predict interlinear glosses and the corresponding morphological segmentation from raw text. We run experiments to determine the optimal way to train models that balance segmentation and glossing accuracy, as well as the alignment between the two tasks. We extend the training corpus of GlossLM and pretrain PolyGloss, a family of seq2seq multilingual models for joint segmentation and glossing that outperforms GlossLM on glossing and beats various open-source LLMs on segmentation, glossing, and alignment. In addition, we demonstrate that PolyGloss can be quickly adapted to a new dataset via low-rank adaptation.