MoirfEolas and CríochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphology

AI summary

Overview

Research area: Natural Language Processing — subword tokenization and its evaluation for morphologically complex, low-resource languages, specifically Irish.

Technical level: Intermediate. The paper assumes familiarity with subword tokenization algorithms (BPE, WordPiece, Unigram LM, SentencePiece) and intrinsic tokenizer metrics, but the core construction methods for both the dataset and the metric are described clearly enough for a reader who understands basic NLP preprocessing.

Scope: A one-sentence scope: the paper builds an Irish morphological dataset (MoirfEolas) and a tokenization evaluation metric (CríochScore), then uses them to score six common tokenization algorithms across multiple vocabulary sizes against Irish morphological boundaries.

What This Paper Is About

Subword tokenizers decide how text gets chopped into units before a language model ever sees it, and for morphologically complex languages like Irish those chops frequently cut across meaningful morpheme boundaries such as eclipses, prefixes and suffixes. The authors build a dedicated Irish morphological resource and a scoring metric so that tokenizers can be measured on how well their segmentations respect Irish morphology, rather than relying only on generic metrics like compression or entropy. The goal is to give Irish NLP developers evidence about which tokenizer algorithms and vocabulary sizes actually line up with the language's structure.

Key Contributions

1. MoirfEolas, an Irish morphological dataset. Over 35,000 Irish words mapped to their eclipses, prefixes and suffixes, built from the Irish side of ParaCrawl V9 (57,587 sentences, 1,447,047 space-separated words) combined with handcrafted Irish affix lists and Irish UniMorph entries. After duplicate removal, the source inventory comprised 330 prefixes, 213 suffixes and 13 eclipses; the final dataset contains 216, 178, and 13 unique prefixes, suffixes and eclipses respectively.

2. CríochScore, a new intrinsic evaluation metric. A metric that scores how well a tokenization aligns with Irish morphological boundaries, giving 1 for a morphological component completely segmented from the stem and not broken further, 0.5 if it is separated but broken into parts, and 0 if it is not separated from the stem. Word-level scores are normalised by the number of evaluated morphological components N in the word, where N ≤ 3, and the overall score is the average of all word-level scores.

3. A comparative evaluation of six tokenization algorithms. HuggingFace implementations of Unigram LM, SentencePiece Unigram LM, WordPiece, BPE, SentencePiece BPE and Byte-level BPE are evaluated at vocabulary sizes of 8k, 16k, 32k and 64k using CríochScore, alongside intrinsic metrics from the literature. SentencePiece Unigram LM was limited to a maximum vocabulary of 43k for this corpus.

4. An adapted affix-level evaluation and absorption analysis for Irish. Prefix-, suffix- and eclipsis-level recall and precision are adapted from prior Turkish work, and an absorption analysis quantifies how often eclipses, prefixes and suffixes are merged into neighbouring tokens or split into fragments.

Main Findings

• The Unigram Language Model aligns with Irish morphology most often. Unigram LM at a vocabulary size of 8k achieved the highest overall CríochScore at 40.55%, with SentencePiece-BPE at 16k close behind at 40.39%. BPE at 64k aligned least often, at 9.6%.

• Smaller vocabularies align better with Irish morphology. Each of the highest CríochScore values reported used a vocabulary size of 8k. The paper attributes this to smaller vocabularies segmenting eclipses, prefixes and suffixes correctly more often than larger ones.

• Larger vocabularies store more whole words and misalign more. Increasing vocabulary size decreased corpus token count, fertility, Rényi Efficiency, and affix- and eclipsis-level precision, while average token length increased. The authors note that greater whole-word storage may harm downstream model generalisability, citing Reddy et al. (2025).

• Unigram segments more, but at a cost to compression. Unigram showed higher fertility and lower average token length than the other algorithms across all sizes, consistent with Brahma et al. (2025)'s hypothesis that morphologically complex languages may need higher fertility. It also had the highest corpus token counts, indicating a trade-off between compression and morphological alignment.

• Unigram has lower Rényi Efficiency than BPE variants. Rényi Efficiency values decreased as vocabulary size increased for all tokenizers, and Unigram's values were lower than the BPE family's, representing a second trade-off between morphological alignment and vocabulary efficiency.

• Prefix boundaries are aligned with more often than suffix boundaries. Although MoirfEolas contains more unique prefixes than suffixes, suffixes appear more frequently in the corpus, which the authors suggest indicates a potential word-initial positional bias in tokenization.

• All eclipsis-only words scored exactly 50%. This applied to the 13 words containing only an eclipsis. On inspection the authors found these words contained an eclipsis paired with a hyphen or apostrophe, so the tokens split at the punctuation; the example given is t-aláram being tokenized as 't', '-', 'aláram'.

• Eclipses behave like noise-level consonant material. Eclipses showed near-perfect recall but near-zero precision, meaning they are almost always present in vocabularies but almost never segmented independently from the stem or prefix. Both SentencePiece implementations, WordPiece 64k and BPE 64k absorb 100% of eclipses into a token with subsequent characters.

• Suffixes are recovered through fragments, not clean suffix tokens. Suffix recall remained high while precision was low, indicating suffixes or parts of them exist in vocabularies without being segmented consistently. Unigram implementations had lower prefix and suffix absorption rates, partly offset by higher suffix split absorption.

• MorphScore was not suitable for Irish. The authors report 2,576 Irish entries in MorphScore versus over 35,000 in MoirfEolas, plus morphologically inaccurate subwords derived from lemma-stem manipulation and a limit of three subwords where Irish morphology in this work requires up to four.

Methodology in Plain English

The authors start with a corpus of Irish text (ParaCrawl V9) and two sources of morphological knowledge: handcrafted lists of Irish eclipses, prefixes and suffixes, plus additional prefix and suffix entries from Irish UniMorph. They merge these sources, remove duplicates, and then use a greedy-longest match method to search for those eclipses, prefixes and suffixes inside each unique word in the corpus. This produces a large mapping of words to their morphological parts. They clean the result by removing English entries using existing Irish morphological analysis tools and by manually checking for incorrect mappings, which removes over 15,000 words.

For evaluation, they define a simple scoring rule: a morpheme gets full credit if it is cleanly separated from the stem as its own token, half credit if it is separated but then broken into further pieces, and no credit if it stays glued to the stem. Each word's score is divided by how many morphological components it was checked for, and a tokenizer's overall score is the average across all words. They then train six different tokenizer algorithms at four vocabulary sizes on the cleaned Irish corpus and score each one. To give context, they also compute standard intrinsic metrics such as corpus token count, fertility, average token length and Rényi Efficiency, and they adapt prefix-, suffix- and eclipsis-level recall and precision from earlier work on Turkish. Finally they run an absorption analysis to see whether morphemes end up merged into neighbouring tokens or split across several tokens.

Why This Matters

Impact on research. The paper addresses a gap the authors identify: intrinsic evaluation of tokenization for Irish had not been investigated before this work, with prior Irish tokenization research being word- and rule-based. It also offers a construction process the authors state can be emulated by other languages to create specialised morphological resources, and it builds on existing morphological metrics such as MorphScore and Suffix Recall.

Real-world applications:

• Irish language technology development, including the transformer and gaBERT-style models discussed in the background, where tokenizer choice was previously described as "the biggest driver of translation performance" and shown to affect downstream performance.
• Machine translation between Irish and other languages, since the paper selects ParaCrawl V9 partly because it supports extrinsic (machine translation) evaluation.
• Reducing the "vocabulary bottleneck" harms for low-resource languages, where tokenization length disparities affect context window usage, cost, processing times and storage.
• Supporting the Digital Plan for Irish (Ní Chasaide et al., 2022), the Irish government's strategy for improving language technologies for the official language.

Industry relevance. Anyone training or deploying language models that handle Irish needs to choose a tokenizer and vocabulary size, and this paper gives concrete comparative evidence across six algorithms and four vocabulary sizes, including explicit trade-offs between morphological alignment, compression and vocabulary efficiency. The authors caution, however, that no algorithm achieved very high CríochScore values and that this does not necessarily imply poor downstream task performance, so the guidance is preliminary.

Future Directions

• Downstream evaluation. The authors state explicitly that downstream evaluation is necessary to verify the practical insights gathered and to establish morphological alignment's role in improving performance.
• Extending the construction process to other languages. The dataset construction and evaluation methodology are designed for Irish but described as easily adapted to other languages with similar morphological forms.
• More granular morphological boundaries. The authors criticise MorphScore for lacking more granular affix boundaries needed in highly agglutinative languages, and their own dataset supports up to four subwords in cases such as t-idirghabhálaí; further work could extend boundary granularity.
• Investigating lenition. Lenition is explicitly not investigated in this work because it mutates the morpheme within a word, leaving it as an open area alongside the eclipsis, prefix and suffix analysis presented here.

Target Audience

Researchers and engineers working on Irish or other low-resource, morphologically complex languages; NLP practitioners selecting tokenizers for multilingual or low-resource model training; and computational morphologists interested in intrinsic tokenization evaluation. Readers evaluating tokenizer quality beyond generic compression and entropy metrics will find the metric design and comparative results directly useful, and those building morphological resources for other languages can reuse the reported construction process.

Authors’ abstract

This paper presents new tokenization resources for Irish and evaluation measures of alignment with the morphological boundaries of the language. We present MoirfEolas, a dataset of over 35,000 Irish words mapped to their respective eclipses, prefixes and suffixes as well as an evaluation metric CríochScore, that evaluates the alignment of tokenizations with the morphological boundaries present in MoirfEolas. We evaluate common tokenization algorithms using CríochScore as well as intrinsic metrics present in the tokenization literature. We find that the Unigram Language Model aligns with Irish morphology more often than the other algorithms evaluated. We also find trade-offs between morphological-alignment of tokenization with both compression as well as vocabulary efficiency, providing practical insights for Irish natural language processing development. This dataset contributes towards combating the Irish language's low-resource status; moreover, the construction process reported in this paper can be emulated by other languages to create specialised morphological resources.