Vocabulary Design and Tokenization Diagnostics

A vocabulary trained mostly on English web text may encode common English words in one token while splitting a low-resource language or a biomedical term into many fragments. Both strings remain representable. But they consume different context and computation.

The gap has been counted rather than supposed. A language written in its own script — Telugu among them — can need up to 5× more tokens to carry the same information. That figure comes from a 2023 study of 22 typologically diverse languages, measured against a live commercial API.

The tokenizer therefore influences latency, truncation, morphology, and the number of steps through which evidence must travel. It is not a neutral formatting layer.

These procedures learn useful inventories, not the one true decomposition of language. The dominant one did not come from linguistics at all. It came from data compression. “We adapt byte pair encoding (BPE) (Gage, 1994), a compression algorithm, to the task of word segmentation,” wrote Sennrich, Haddow and Birch at Edinburgh in 2016. The payoff they measured over a back-off dictionary baseline was up to 1.1 and 1.3 BLEU, on the WMT 15 English→German and English→Russian tasks. Jurafsky and Martin now record where that borrowed heuristic ended up: “Two tokenization algorithms are widely used in modern language models: byte-pair encoding (BPE) (Sennrich et al., 2016), and unigram language modeling (ULM) (Kudo, 2018).” So the frequency-driven merge process in the first column and the pruned probabilistic segmentation model in the third are the two survivors. One of them is a 1994 data-compression trick with 1.1 to 1.3 BLEU attached.

Byte-aware subwords are a budget decision with published arithmetic. A BPE built over Unicode code points would need a base vocabulary of over 130,000 entries before a single multi-symbol token is added. Typical BPE vocabularies hold 32,000 to 64,000. The GPT-2 report gives the alternative in one line: “In contrast, a byte-level version of BPE only requires a base vocabulary of size 256.” GPT-2's finished vocabulary is 50,257 tokens, the value Hugging Face still ships as the GPT2Config default. That is what “avoids unknown characters” costs: 256 base symbols instead of 130,000, inside a 50,257-token budget.

What those 256 symbols do not buy is an equal price per character. The same column warns that byte fallback “can lengthen unusual text”. UTF-8 is why. It is a variable-length encoding, and RFC 3629 is explicit about the range: “In UTF-8, characters from the U+0000..U+10FFFF range (the UTF-16 accessible range) are encoded using sequences of 1 to 4 octets”. The 128 ASCII characters cost one byte. Code points at or above 128 cost two, three or four. A byte-level base alphabet therefore spends between one and four positions per character before any merge is applied. Which end of that range a reader pays is decided by the script they write in.

Changes at any stage can make old data or models incompatible.

Stages three and four are visible in any released checkpoint, and they rarely match the paper. The BERT paper specifies a round number: “We use WordPiece embeddings (Wu et al., 2016) with a 30,000 token vocabulary.” The artifact that shipped is not that number. google-bert/bert-base-uncased declares vocab_size 30522, and its vocab.txt runs to 30,522 lines. Of those, 994 are [unused0] through [unused993] — empty slots deliberately left for later additions. Five more are control symbols nailed to fixed integers: [PAD]=0, [UNK]=100, [CLS]=101, [SEP]=102, [MASK]=103, with [PAD] first in the file and the other four at lines 101–104.

That is stage four in one file. A design figure of 30,000 became an address space of 30,522 entries. 994 of them are reserved emptiness. Five are pinned to integers that every downstream serving path, every cached dataset and every trained embedding row now agrees on.

Stage five is not ceremonial either. Land and Bartolo found 2,450 confirmed under-trained tokens among 2,966 tested in Qwen1.5 32B's 151,646-token vocabulary. A shipped model carried that defect until somebody audited for it.

Average tokens per sentence is too coarse to explain who pays the cost.

The size of the gap has been measured rather than estimated. A NeurIPS 2023 paper found that “the same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases”, and that “these disparities persist even for tokenizers that are intentionally trained for multilingual support”. Character-level and byte-level models narrow the gap without closing it: such models “also exhibit over 4 times the difference in the encoding length for some language pairs”. A fifteen-fold ratio is a fifteen-fold difference in the cost of a commercial language service. It is also a difference in processing time, and in how much of a document fits inside the context window. It is paid by the speakers of one language and not the other.

The second thing an audit finds is entries that should not be in the vocabulary at all. Land and Bartolo, at Cohere, defined the class in 2024 as “tokens present in the tokenizer vocabulary but that are nearly or entirely absent during model training”. Then they counted them. 67 of 999 candidate tokens in GPT-2 XL, whose vocabulary holds 50,257 entries. 200 of 999 in GPT-J 6B. And 2,450 of 2,966 in Qwen1.5 32B, whose vocabulary holds 151,646. The Gemma team released a fix in response to their report.

The failure mode had been public since 5 February 2023, when Rumbelow and mwatkins described anomalous tokens such as SolidGoldMagikarp and RandomRedditorWithNo that made GPT-2 and GPT-3 evade, hallucinate or emit garbled output: “Many of these tokens reliably break determinism in the OpenAI GPT-3 playground at temperature 0 (which theoretically shouldn't happen).” Two years elapsed between that post and the systematic count. A vocabulary is not audited by being read.

Use real text and deliberately difficult examples rather than a few attractive demonstrations.

Step five is what makes the first four interpretable. The 68% figure could be pinned on the tokenizer only because the tokenizer varied across 24 trained models while everything else stood still. An audit that changes the vocabulary and the corpus at the same time produces a number nobody can assign to either. And because that same work found fertility and parity unreliable as predictors of downstream performance, step two cannot stand in for step five. A favourable fertility table is a hypothesis about quality. It is not a measurement of it.

Step four has a concrete acceptance test. Decode what you encoded, then confirm that the reserved symbols still sit where the shipped vocabulary file says they sit — in bert-base-uncased, [PAD] at 0 and [UNK], [CLS], [SEP] and [MASK] at 100 to 103.

Token IDs index learned embedding rows, so a new vocabulary cannot normally be substituted beneath an existing model without adaptation. Added tokens have to be initialized and trained. Removed or reordered IDs can corrupt every embedding lookup.

The 994 [unused0] through [unused993] entries in bert-base-uncased are what foresight about this looks like. 994 addresses set aside in a 30,522-entry vocabulary, so that a later addition can be given an ID without renumbering the entries that were already trained. A vocabulary shipped without that provision has no such room. Every insertion is a renumbering.

Plan tokenizer upgrades with model retraining or controlled adaptation, compatibility tests, data versioning, and serving rollout; treat the old tokenizer as an artifact needed for historical reproducibility.

Use original documents rather than already tokenized examples; report characters and words per token, truncation, sequence percentiles, fallback, latency, and task quality by language. Include at least one language written in its own script. That is where the up-to-5× token requirement was found, across 22 languages measured on a live commercial API. Price the workload in currency as well as in tokens.

Inspect twenty high-cost examples and explain whether fragmentation reflects domain mismatch, script coverage, normalization, or an acceptable trade-off. Then check the vocabulary itself rather than only its output: reserved symbols at their declared IDs, round-trip decoding, and any entries the model may never have trained on — the class Land and Bartolo confirmed at 67 of 999 candidate tokens even in GPT-2 XL.