The Pulse
Hugging Face Claims a 50× BERT Tokenization Speedup
Hugging Face says changes to its Tokenizers library make BERT text encoding roughly 50 times faster than version 0.23 in its benchmarks. The optimization is still at pull-request stage, and the largest gains come from ASCII-heavy text rathe
AI.info Team · Written with AI ·
Hugging Face’s claimed 50× speedup for BERT tokenization comes with a qualification: the benchmark combines gains already made in its Tokenizers library with a new optimization that is still in pull-request review. The new work adds up to another 5.7× for BERT’s uncased tokenizer, the company says, while performance varies sharply by script and input length.
Simon Brandeis, author of the Hugging Face post, writes: “At the time of writing, the changes are still in Pull Request stage.” The distinction matters for users deciding whether to expect the full headline gain in a released package—or in their own workloads.
The 50× figure combines two rounds of work
The post compares bert-base-uncased with Tokenizers 0.23. Before the latest proposed changes, the model’s encoding was already about 8.6× faster, according to Brandeis. The new normalization work adds up to 5.7× over that newer baseline, bringing the combined improvement to about 50× over version 0.23.
Those figures describe single-thread benchmark results, not a guaranteed improvement in application response time. Hugging Face measured on two AMD EPYC 7R13 hosts and an Intel Xeon Platinum 8488C, pinning each run to one core. Each measurement repeatedly encoded a 5–6 MB text corpus for five seconds; the author says the speedup ratios agreed within a few percent across the machines.
Normalization took more time than choosing tokens
The optimization targets work that happens before WordPiece selects tokens. In a profile of a warm English encoding pass with bert-base-uncased, normalization accounted for 88% of encoding time; WordPiece accounted for 8%. The normalizer cleans text, handles Chinese characters, decomposes Unicode, removes accent marks and lowercases letters.
Hugging Face’s changes aim to skip or reduce that work when the input allows it. The code checks ASCII bytes in groups of 16, avoiding Unicode processing for ordinary ASCII runs. For other scripts, it caches normalization decisions for pages of 256 nearby code points. If the input is already in the required form, the normalizer can return it without allocating a new string; when changes are needed, it can copy untouched sections in bulk.
The reported results preserve token IDs across the benchmark corpora. English and source code see the strongest gains from skipping ASCII processing. Russian, Greek and Arabic text—each about 11% ASCII in the tested corpora—improved by 2.2× to 3.2×, with the page cache accounting for those gains.
Chinese and short inputs show the limits
The speedup does not carry evenly across languages. The post says Chinese text gains little because BERT inserts spaces around Chinese characters, leaving little ASCII work to skip. For ModernBERT, which uses NFC normalization, English encoding barely changes at 1.06×, while code, Arabic and Russian improve by about 2×.
Small inputs can also lose time to the new machinery. In normalization-only tests, 55 of 320 measurements were slower beyond the stated noise range; inputs of 32 bytes or fewer could regress, with an 8-byte Chinese input reaching 0.35× for accent stripping. Cased BERT normalization on Chinese was 8% to 15% slower for inputs of 64 bytes or more.
For teams evaluating the change, the practical question is whether their text resembles the benchmark’s English-heavy, multi-megabyte corpora. Hugging Face’s results point to profiling real input by language and length, then checking both throughput and correctness. The proposed code was still under review when Brandeis published the measurements on October 15, 2026.