Unicode, Normalization, and the Boundaries of Text

These layers explain why character counts, offsets, and display can disagree.

The bottom layer has a specification and a date. RFC 3629, from November 2003, defines UTF-8 and confines it to the range U+0000 to U+10FFFF: “In UTF-8, characters from the U+0000..U+10FFFF range (the UTF-16 accessible range) are encoded using sequences of 1 to 4 octets”. The first octet of a multi-octet sequence says how many octets are in the sequence. One code point is therefore anywhere between one and four bytes.

The layer above it is still not the reader's layer. Unicode calls a user-perceived character a grapheme cluster, and treats the cluster as the programmatic approximation of it. Unicode Standard Annex #29 gives the smallest possible demonstration: “For example, “G” + grave-accent is a user-perceived character: users think of it as a single character, yet is actually represented by two Unicode code points.” The W3C supplies two more from other writing systems. The Vietnamese ề is 3 code points. The Bangla ksha is 3 code points. Byte length, code-point count and what a reader perceives as a character are three separate numbers. A system that uses one where it means another has a bug it will not see until the text stops being English.

The top layer, shaping and rendering, has failure modes of its own, and one of them reached a security bulletin. iOS 11.2.6 shipped on February 19, 2018 with a CoreText fix, credited to an anonymous researcher, under CVE-2018-4124. Apple's impact line is one sentence: “Impact: Processing a maliciously crafted string may lead to heap corruption”. NIST's NVD entry, published 2018-04-03, says what the string contained: a certain Telugu character, which let remote attackers cause a denial of service in iOS before 11.2.6, macOS before 10.13.3 Supplemental Update, tvOS before 11.2.6 and watchOS before 4.2.3. One cluster, in one script, took down four shipping operating systems — at a layer that never appears in a tokenizer's unit tests.

Each transformation is reversible to a different degree, and carries a different risk profile.

There are exactly four Unicode normalization forms, and no more. NFD is canonical decomposition. NFC is canonical decomposition followed by canonical composition. NFKD is compatibility decomposition, and NFKC is compatibility decomposition followed by canonical composition. The K is the letter to watch. Canonical equivalence is the strong relation: it holds between sequences that represent the same abstract character and that should be treated and displayed as identical. Compatibility equivalence is deliberately weaker. It can hold between characters whose visual appearance or behaviour differs — superscripts, circled letters, full-width forms.

That NFKC is a matching aid and not a storage format is not this lesson's editorial opinion. It is printed in Unicode Standard Annex #15, the annex that defines the forms: “Normalization Forms KC and KD must not be blindly applied to arbitrary text. Because they erase many formatting distinctions, they will prevent round-trip conversion to and from many legacy character sets, and unless supplanted by formatting markup, they may remove distinctions that are important to the semantics of the text.” Note what is being conceded. The loss is not a bug in an implementation. It is the definition working correctly.

A second standards body wrote the same warning into a protocol. The IETF's PRECIS framework, RFC 8264 of October 2017, sets the default for strings compared on the wire: “In accordance with [RFC5198], Normalization Form C (NFC) is RECOMMENDED”. It warns in the same section that “certain Unicode normalization forms, especially NFKC and NFKD, can result in significant loss of information in various circumstances”. One body defines the forms, the other specifies how strings are compared; they chose the same default and printed the same caution. Casing, cleaning and transliteration sit outside both documents entirely, which is precisely why they need a policy of their own.

The safest policy is rarely “strip everything unusual.” Each of the first three cases below has a CVE number, a vendor document or a Unicode data file behind it. That is the difference between a preprocessing rule you can defend and one you inherited.

Extraction systems often return character spans that another application highlights in the original document. If normalization changes string length, a span measured on processed text points to the wrong location. The units are not interchangeable either. The Vietnamese ề from the W3C's worked example is 3 code points and a single perceived character, so a span counted in one unit and applied in the other is silently wrong rather than loudly broken.

A robust pipeline either preserves a reversible alignment map or predicts spans against a stable representation. Raw documents, normalized views, and token offsets need explicit versioned relationships. The version matters as much as the map. NormalizationTest-17.0.0.txt and SpecialCasing-17.0.0.txt both carry a Unicode version in their names, and an offset computed under one release of the data files is a claim about that release.

A dozen carefully chosen strings can expose more risk than millions of ordinary examples. For normalization itself, tens of thousands of them have already been chosen and published.

Unicode ships the file. NormalizationTest-17.0.0.txt, dated 2025-06-30, contains 20,034 test lines in six parts: Part0 specific cases (45), Part1 character-by-character (17,086), Part2 canonical order (1,936), Part3 PRI #29 (194), Part4 canonical closures excluding Hangul (735) and Part5 chained primary composites (38). Each line gives five columns — source; NFC; NFD; NFKC; NFKD. The CONFORMANCE section of the header states the terms: “1. The following invariants must be true for all conformant implementations”.

That file is not a document to admire from a distance. CPython's own test suite downloads it and asserts c2 == NFC(c1) == NFC(c2) == NFC(c3), line by line, together with the parallel NFD, NFKC and NFKD invariants. That is what CPython treats as the floor, not the ceiling.

So step 1 below begins as a download rather than an act of invention. The equivalence pairs you still have to write yourself are the ones the Consortium cannot know: your users' names, your identifiers, your markup, your locales. The tr and az casing rules. The ZWJ sequences your product actually receives. The directional controls your ingest is about to strip. Those go into the same harness, and they run on every release of the data files, not once.

A production policy should name encoding assumptions, normalization form, markup handling, casing, whitespace rules, URL treatment, and invalid-input behavior. Each rule needs a reason tied to a task or safety requirement.

What it costs to leave one of those unwritten has a date and a company attached. In June 2013 Spotify accounts were hijacked through the password-reset flow, one of them a support-forum manager's, and the company disabled account creation for hours. Spotify's engineering blog published the account on 18 June 2013. Its worked example is the username ᴮᴵᴳᴮᴵᴿᴰ — U+1D2E U+1D35 U+1D33 U+1D2E U+1D35 U+1D3F U+1D30 — which twisted's XMPP nodeprep.prepare canonicalized to 'BIGBIRD' on the first application and 'bigbird' on the second. The post says explicitly that the account 'bigbird' was not among the attacked accounts; it was picked as an example name. Mikael Goldmann's diagnosis is one sentence: “We were relying on nodeprep.prepare being idempotent, and it wasn’t.” Registration reopened behind a hand-rolled fixpoint check, X == canonical_username(X), and the permanent fix rejected any name where old_prepare(old_prepare(name)) != old_prepare(name).

The root cause sat upstream of Spotify, in an unwritten rule of exactly the kind this section asks for. The nodeprep draft fixes its input and output repertoire at Unicode 3.2 but never tells implementers to validate the input. The ᴮᴵᴳᴮᴵᴿᴰ code points are outside Unicode 3.2. The IETF eventually reached the same verdict about the whole framework and replaced Stringprep with PRECIS — first published as RFC 7564 in 2015, current version RFC 8264, October 2017 — giving the reason in its own words: “most significantly, Stringprep was tied to Unicode version 3.2”. A canonicalizer is a versioned dependency. Idempotence under it is a property to be tested, not assumed.

This detail makes training and inference consistent. It also lets future teams reverse a harmful choice instead of reconstructing it from scattered code.