Tokens, Tokenizers, and the Boundaries a Model Sees

The same text, translated, can cost fifteen times more to send to a model. That was measured, not estimated. Four researchers took identical text in several languages and counted what each version cost in tokens. The paper is called "Language Model Tokenizers Introduce Unfairness Between Languages", and it appeared at NeurIPS in 2023. Its abstract states the spread without hedging: “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.”

The obvious remedies do not close the gap. The same abstract continues: “These disparities persist even for tokenizers that are intentionally trained for multilingual support. Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs.”

A limit expressed in tokens is therefore not one limit. It is a different limit per language, hidden behind a single number in the documentation. Two users can send the same document, to the same product, under the same stated context window. Only one of them gets truncated.

A tokenizer maps text or other serialized input into discrete IDs from a vocabulary. Common systems use subword pieces, bytes, or combinations that balance vocabulary size, sequence length, and coverage. The resulting pieces may align with a word, part of a word, punctuation, whitespace, or raw bytes. A token boundary does not prove a semantic boundary. The same visible text can tokenize differently across models.

Numbers are where that stops being an abstraction, because there the boundary changes the answer. The field does not even agree on where a number ends. Singh and Strouse open their 2024 paper on arithmetic in frontier models by listing the disagreement: “With the increased use of LLMs for reasoning, various number-specific tokenization schemes have been adopted, with popular models like LLaMa and PaLM opting for single-digit tokenization while GPT-3.5 and GPT-4 have separate tokens for each 1-, 2-, and 3-digit numbers.”

The chunking is not cosmetic. Comma-separating numbers at inference time forces right-to-left tokenization, and Singh and Strouse report that this raises arithmetic accuracy by up to 20%. The arithmetic did not change and the model did not change. Only the boundaries did.

No tokenizer is uniformly best across languages, code, noise, and deployment constraints. Each of the three approaches has a published measurement behind it.

Subword vocabulary. Frequent strings become compact reusable pieces, so the vocabulary encodes corpus frequency. What the training data said often is cheap. Rare forms fragment. The cost is measurable and it is per language: over the mC4 pre-training corpus, mT5's subword tokenizer compresses at 4.1 bytes per SentencePiece token overall. That average hides a range from 2.5 bytes per token for Maltese to 9.0 for Khmer.

Byte-oriented representation. ByT5 threw the subword vocabulary away. Its 2022 paper says what replaced mT5's SentencePiece pieces: “The bytes are embedded to the model hidden size using a vocabulary of 256 possible byte values.” The remainder of the vocabulary is three entries. “An additional 3 IDs are reserved for special tokens: padding, end-of-sentence, and an unused <UNK> token that we include only for convention.” No input is unrepresentable. The sequences get longer.

Domain-specialized vocabulary. Compactness inside one field is paid for with compatibility outside it. SciBERT built SCIVOCAB — a 30K WordPiece vocabulary trained with SentencePiece over 1.14M Semantic Scholar papers, 3.17B tokens, 18% computer science and 82% biomedical — and set it against BERT's general-domain BASEVOCAB. Two vocabularies of the same size, built the same way, on different text. The 2019 paper reports what came out: “The resulting token overlap between BASEVOCAB and SCIVOCAB is 42%, illustrating a substantial difference in frequently used words between scientific and general domain texts.” Migrating between them is not a preprocessing change. It changes which strings are cheap and which embeddings mean anything.

A product should record each transformation before the model sees an ID sequence. Raw bytes arrive from files, forms, APIs, or user devices. An encoding such as UTF-8 turns them into code points. The product may normalize the text and insert roles, separators, or tool schemas. The deployed vocabulary maps that serialized input to discrete IDs. Context packing then decides which tokens survive truncation and how the remaining space is allocated.

The normalization stage is a specified step, not a local convention. Unicode Standard Annex #15, "Unicode Normalization Forms", defines exactly four: NFD, NFC, NFKD and NFKC. RFC 5198 named the one expected on the wire back in 2008: “Before transmission, all character sequences SHOULD be normalized according to Unicode normalization form "NFC" (see Section 3).”

When two services in the same pipeline pick different forms, or one picks none, the same visible string reaches the tokenizer as different code points and leaves it as different IDs. The stage that made the difference is upstream of everything the model logs.

Use production-like data rather than a handful of demonstration prompts. Sample real inputs across languages, code, identifiers, OCR noise, and long documents. Measure the distributions: tokens per character, per word, and per task component. Inspect fragmentation to find the names, terms, and scripts with extreme token counts. Test truncation policies by measuring which facts disappear under realistic packing. Then version the tokenizer, tying counts, prompts, embeddings, and regressions to the exact release.

The measurement step has a published worked example. Seven researchers ran it over OpenAI's API for 22 typologically diverse languages in 2023, in a paper titled "Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models". Some languages need 5 times as many tokens to convey the same information. On XLSUM, the combined prompting and generation cost ran up to 4x higher in Telugu and Amharic. The paper puts the finding in the currency a user actually pays in: “While LMs like ChatGPT might perform tasks in Telugu, for example, a user in Andhra Pradesh might pay 5× more than an English user in the US for an equivalent use of the model.”

An audit that reported a single average tokens-per-character figure would have surfaced none of that. The spread across the 22 languages is the finding. The mean is what hides it.

Embeddings are learned for token IDs, attention operates over token positions, and serving cost grows with tokenized sequences. Retrieval systems depend on token-aware chunking and context packing too. Each measurement above lands in a different layer. The normalization form decides which code points arrive. The vocabulary decides how many pieces they become. The 0.1–1% of severely under-trained entries decides whether some of those pieces mean anything. The digit boundaries decide whether a sum comes out right.

Treat tokenization as part of the application contract. A tokenizer change is not merely a preprocessing refactor when it changes cost, truncation, or model behavior. The published record shows it changes all three.