The Pulse
Microsoft Tests a Morphology-Aware Tokenizer Across 104 Languages
Microsoft researchers introduce the Latent Core Tokenizer, a 200,000-token system evaluated across 104 languages. Their paper reports better morphological alignment and lower tokens-per-word than several common tokenizers, while noting limi
AI.info Team · Written with AI ·
Fewer tokens per word, with boundaries that track morphology
At a 200,000-token vocabulary, Microsoft’s Latent Core Tokenizer averaged 1.999 tokens per word, compared with 2.241 for Byte Pair Encoding, or BPE. The result comes from a study spanning 104 languages, posted on October 8, 2026, that tests whether a tokenizer can compress text while better preserving meaningful word parts. The authors report stronger morphological alignment than BPE, Unigram and parity-aware BPE, alongside similar cross-language disparities in tokenization cost to BPE.
The researchers describe their central finding this way: “Our findings show that compression alone does not predict representation quality and highlight the importance of morphology-driven structural discovery and how frequency is used to allocate the final vocabulary across languages.” Felermino D. M. A. Ali et al., Microsoft Research
The work introduces a research method, not a new Microsoft product or a claim that models already perform better in every language. Its focus is the tokenizer—the component that breaks text into units a language model can process. Those units affect sequence length and how a fixed vocabulary is shared among languages with different scripts, word structures and amounts of available text.
LCT separates language analysis from token creation
Conventional subword tokenizers generally discover recurring pieces and select the final vocabulary in one process. The Latent Core Tokenizer, or LCT, separates those steps. It first analyzes text for reusable internal structure, then builds the vocabulary that the model actually receives; its internal pieces guide that construction but are not supplied as extra model inputs.
For structural discovery, LCT combines a Minimum Description Length method with branching-entropy signals and morphotactic constraints. Those signals help identify likely boundaries and distinguish prefix-, stem- and suffix-like behavior without requiring gold-standard morphological labels. The method then packs supported pieces into surface tokens, merging combinations according to their contribution to token savings.
The team also adjusts how language frequency informs the process. Instead of letting the largest text collections dominate vocabulary choices, LCT can rebalance evidence across languages; its experiments compare different degrees of that balancing. The authors say this is not an equal quota of vocabulary entries for each language: a piece shared across languages can still receive support from all of them.
The gains depend on what the tokenizer is measured against
On the paper’s intrinsic tests at a 200,000-token vocabulary, LCT’s LCT-MEM configuration reached a MorphScore F1 of 0.5209, against 0.4410 for BPE. MorphScore measures how well token boundaries align with morpheme boundaries. The fertility comparison—tokens per word—uses UDHR translations covering 93 languages, while MorphScore covers 65 languages with sufficient annotated examples, rather than all 104 languages in the tokenizer study.
For downstream tests, the authors trained DeBERTa-v3-style XSmall encoders and evaluated them on XNLI, Belebele, XStoryCloze and PAWS-X. LCT-MEM with partial language balancing improved the reported aggregate score by 1.48 points over BPE, 1.83 over Unigram and 2.00 over parity-aware BPE. The paper’s authors also report that the shortest sequences did not produce the strongest downstream results, a distinction that complicates any attempt to judge tokenizers by compression alone.
One pretraining run leaves a key uncertainty
The authors flag limits on how far to generalize the results. Each encoder was pretrained once, so the reported variance reflects five fine-tuning runs, not repeated pretraining; the paper says some of the 1.48-point gain over BPE could reflect pretraining variation. The experiments also use only the small DeBERTa-v3-style encoder, so they do not establish whether the result carries over to larger models or decoder-only systems.
Coverage is uneven, too: 62 of the 79 languages represented in the downstream evaluation appear only in Belebele, making the aggregate findings heavily dependent on that benchmark. In the smallest pretraining-data quartile, LCT-MEM’s Belebele result was essentially tied with BPE, the authors report. Whether the tokenizer’s gains persist across repeated pretraining runs, broader task coverage and larger model architectures remains an open test.