Segmentation: Sentences, Words, Subwords, and Bytes

Boundary decisions that look trivial in English become explicit engineering choices in real corpora.

Unicode says so in its own specification, rather than leaving implementers to discover it. Annex #29 defines the default boundaries for grapheme clusters, words and sentences, and it warns readers directly: “Word boundaries are not restricted to whitespace and punctuation. Indeed, some languages do not use spaces at all”. It is more specific a line later: “reliable detection of word boundaries in languages such as Thai, Lao, Chinese, or Japanese requires the use of dictionary lookup or other mechanisms”. ICU, the reference implementation, ships precisely that. Its own guide records: “Some languages are written without spaces, and word and line breaking requires more than rules over character sequences. ICU provides dictionary support for word boundaries in Chinese, Japanese, Thai, Lao, Khmer and Burmese”.

How large a gap that leaves has been measured on the record. In the CoNLL 2018 shared task, 26 systems parsed raw text into Universal Dependencies across 82 treebanks in 57 languages. In aggregate, word segmentation looked finished. Uppsala took the best macro-averaged word F1 at 98.18, and even the BASELINE UDPipe 1.2 reached 96.97. Sentence segmentation did not follow. The best score was 83.87, from HIT-SCIR, against a baseline of 83.01. The winning system barely separated itself from the untuned starting point, on the task with far more error left in it.

The aggregate also hides where the failure actually lives. On Thai (th_pud) the best word-segmentation F1 was 69.93, and the cross-system average was 17.16 ± 20.57. On Japanese “Modern” (ja_modern) the best was 75.69, the average 59.40 ± 7.70.

The organisers did not treat this as a surprise. “Not surprisingly, word segmentation is difficult for the low-resource languages and for languages like Chinese, Vietnamese, Japanese and Thai, where spaces do not separate words.”

A tokenizer that splits on spaces is not a simple tokenizer. It is a tokenizer whose headline number is 98.18 and whose Thai number is 17.16. Only one of those two ever appears in a summary table.

A pipeline may use several levels at once rather than choosing one forever.

The levels below are not a ladder of increasing correctness. Each one is a policy that somebody had to write down and defend. The same corpus can carry all of them at once: a stable document identifier, a 100-word retrieval block, a sentence split whose accuracy depends on the source, and a 256-value byte alphabet underneath. The sections that follow attach a measured number to three of those choices.

Different units trade sequence length, linguistic alignment, robustness and vocabulary size. Each side of that trade has been measured, so it can be quoted rather than asserted.

On the subword side the disparity is large, and it falls unevenly across languages. Petrov and colleagues put it in the abstract of their 2023 NeurIPS paper: “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.” A second group reproduced the effect independently that year, across 22 typologically diverse languages, and found “some languages requiring 5 times as many tokens than others”. Ahia and colleagues also estimated that a Telugu user could pay 5× more than an English user for equivalent use of the same API. Sequence length is not a neutral property of the encoder. It is an invoice, and it is not addressed to everyone equally.

Dropping to a smaller alphabet does not cancel the effect. The same NeurIPS paper records that “Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs.”

ByT5, published in 2022, prices the byte column exactly. It feeds UTF-8 bytes directly, with “a vocabulary of 256 possible byte values” plus 3 reserved IDs. That collapses the vocabulary side of the parameter budget: vocabulary-related parameters fall from 85% of the model in mT5-Small and 16% in mT5-XXL to 0.3% and 0.02%. The bill arrives as time. ByT5 runs 1.5 to 2.6 times slower than mT5 on Dakshina transliteration and 3.7 to 6.4 times slower on GEM-XSum summarization. On XNLI zero-shot classification the paper reports a model that “is 6.4 to 9.5 times slower”. Its own introduction says why: “The main drawback of byte-level models is that byte sequences tend to be significantly longer than token sequences.”

So the four columns below are not four flavours. They are four points on one curve. It runs from 85% of a model spent on a vocabulary down to 0.02%, and from one inference-time unit up to 9.5.

Breaking a long document into passages can improve retrieval and fit model limits. But a claim may depend on a definition in an earlier section. Overlap helps at boundaries, and it also creates duplicate evidence and additional cost.

What such a policy looks like in practice is on the record, down to the unit count. For Dense Passage Retrieval, in 2020, Karpukhin and colleagues took the December 20, 2018 English Wikipedia dump and cleaned it of tables, infoboxes, lists and disambiguation pages. Then, in their pre-processing section: “We then split each article into multiple, disjoint text blocks of 100 words as passages, serving as our basic retrieval units, following (Wang et al., 2019), which results in 21,015,324 passages in the end.” Each block was prepended with its article title and an [SEP] token. One round number, chosen once, fixed the unit of evidence 21,015,324 times.

And the choice did not stay inside that paper. The Retrieval-Augmented Generation paper reused the identical split later the same year: “Each Wikipedia article is split into disjoint 100-word chunks, to make a total of 21M documents.” That was the non-parametric knowledge source for every experiment in it. Disjoint means no overlap. Every dependency that crossed one of those 21 million cuts was simply not represented, in both systems, unexamined.

Preserve stable document and position identifiers so passages can be reassembled. Evaluate facts and references that cross chunk boundaries rather than assuming local context is sufficient.

Use both direct boundary measures and downstream tests. Re-run them whenever the source changes, because a segmenter's published accuracy belongs to the corpus it was published on.

Punkt is the case worth knowing, and it is instructive because it began so well. Kiss and Strunk published it in 2006, “tested extensively on eleven different languages” in their own abstract's words. The mean sentence-boundary error rate across those eleven newspaper corpora was 1.26%. That beat the supervised MxTerminator's 1.77%, even though MxTerminator had been given as many as 34,256 training instances. On the Brown corpus the error rate was 1.02%. On The Works of Edgar Allan Poe, 0.80%. Their introduction explains where the difficulty comes from: “Graphemes often serve more than one purpose in writing systems. The period, which is employed as sentence boundary marker, is no exception. It is also used to mark abbreviations, initials, ordinal numbers, and ellipses.”

Nineteen years later, the same splitter met a different kind of text. A 2025 evaluation ran it over five legal datasets: 25,000+ documents, 197,000+ annotated sentence boundaries. NLTK's Punkt measured 62.1% precision on legal text, against 91.1% for the domain-retrained NUPunkt. That is a 29–32% precision gap, opened by nothing but a change of source. About 1% error on newswire and 62.1% precision on legal documents are the same program, unmodified, under two evaluations.

So step 1 is not a formality, and step 4 is not optional. Sample the sources you will actually run on. Annotate the boundary this product needs. Separate missed boundaries from harmful extra splits. Then check whether those local errors move retrieval, extraction, translation, latency and sequence length at all.

Take a set of long customer conversations. Create one policy for intent routing and another for evidence retrieval, then justify the document, turn, passage and overlap choices.

Estimate token lengths, duplicated text, cross-boundary references and source reconstruction. The two tasks may need different segmentation even though they use the same raw messages. Write the numbers down the way Dense Passage Retrieval did — 100 words, disjoint, title prepended, 21,015,324 units — so that a later reader can see what was chosen rather than infer it.

Then ask the second question, the one the Retrieval-Augmented Generation paper answers by example. If someone reuses your policy unchanged for a different task, what breaks, and would either of you notice?