# Hugging Face Claims a 50× BERT Tokenization Speedup

Published 2026-10-15 · AI.info Team · AI.info

> Hugging Face’s benchmarks show changes to its Tokenizers library making BERT text encoding roughly 50 times faster than version 0.23. The optimization is still at pull-request stage, and the largest gains come from ASCII-heavy text rather than Chinese or very short inputs.

[Hugging Face](https://huggingface.co/blog/huggingface/tokenmaxxing-bert)’s claimed 50× speedup for BERT tokenization comes with a qualification: the benchmark combines gains already made in its Tokenizers library with a new optimization that is still in pull-request review. The new work adds up to another 5.7× for BERT’s uncased tokenizer, the company says, while performance varies sharply by script and input length.

Simon Brandeis, author of the Hugging Face post, writes: “At the time of writing, the changes are still in Pull Request stage.” The distinction matters for users deciding whether to expect the full headline gain in a released package—or in their own workloads.

## The 50× figure combines two rounds of work

The post compares `bert-base-uncased` with Tokenizers 0.23. Before the latest proposed changes, the model’s encoding was already about 8.6× faster, according to Brandeis. The new normalization work adds up to 5.7× over that newer baseline, bringing the combined improvement to about 50× over version 0.23.

Those figures describe single-thread benchmark results, not a guaranteed improvement in application response time. Hugging Face measured on two AMD EPYC 7R13 hosts and an Intel Xeon Platinum 8488C, pinning each run to one core. Each measurement repeatedly encoded a 5–6 MB text corpus for five seconds; the author says the speedup ratios agreed within a few percent across the machines.

## Normalization took more time than choosing tokens

The optimization targets work that happens before WordPiece selects tokens. In a profile of a warm English encoding pass with `bert-base-uncased`, normalization accounted for 88% of encoding time; WordPiece accounted for 8%. The normalizer cleans text, handles Chinese characters, decomposes Unicode, removes accent marks and lowercases letters.

Hugging Face’s changes aim to skip or reduce that work when the input allows it. The code checks ASCII bytes in groups of 16, avoiding Unicode processing for ordinary ASCII runs. For other scripts, it caches normalization decisions for pages of 256 nearby code points. If the input is already in the required form, the normalizer can return it without allocating a new string; when changes are needed, it can copy untouched sections in bulk.

The reported results preserve token IDs across the benchmark corpora. English and source code see the strongest gains from skipping ASCII processing. Russian, Greek and Arabic text—each about 11% ASCII in the tested corpora—improved by 2.2× to 3.2×, with the page cache accounting for those gains.

## Chinese and short inputs show the limits

The speedup does not carry evenly across languages. Chinese text gains little because BERT inserts spaces around Chinese characters, leaving little ASCII work to skip, the post says. For ModernBERT, which uses NFC normalization, English encoding barely changes at 1.06×, while code, Arabic and Russian improve by about 2×.

Small inputs can also lose time to the new machinery. In normalization-only tests, 55 of 320 measurements were slower beyond the stated noise range; inputs of 32 bytes or fewer could regress, with an 8-byte Chinese input reaching 0.35× for accent stripping. Cased BERT normalization on Chinese was 8% to 15% slower for inputs of 64 bytes or more.

For teams evaluating the change, the practical question is whether their text resembles the benchmark’s English-heavy, multi-megabyte corpora. Hugging Face’s results point to profiling real input by language and length, then checking both throughput and correctness. The proposed code was still under review when Brandeis published the measurements on October 15, 2026.

## Sources

1. Original source · Blog post [Hugging Face](https://huggingface.co/blog/huggingface/tokenmaxxing-bert) huggingface.co

## Related stories

- [Hugging Face’s Tokenizers v1 reaches 30x faster encoding](https://ai.info/news/hugging-face-tokenizers-v1-30x-faster-encoding)
- [Hugging Face switches language-model routing to OpenRouter](https://ai.info/news/hugging-face-openrouter-inference-routing)
- [State of AI Report Details Agents’ Breach of Hugging Face](https://ai.info/news/state-of-ai-report-agents-hugging-face-breach)
- [Microsoft and Hugging Face Test Agents Across 507 Workflows](https://ai.info/news/microsoft-hugging-face-thinkingbox-agent-reliability-benchmark)
- [DOJ Questions Nvidia’s $12.93B Hugging Face Deal](https://ai.info/news/doj-questions-nvidia-hugging-face-acquisition)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/hugging-face-bert-tokenization-50x-speedup
