The Pulse
Hugging Face’s Tokenizers v1 reaches 30x faster encoding
Hugging Face says its Tokenizers v1 release candidate encodes text three to 30 times faster than v0.23 on an Apple M4 Max. The redesign preserves token IDs while replacing regex splitting, repeated allocations and lock contention with SIMD
AI.info Team ·
Hugging Face’s Tokenizers v1 encodes text between three and 30 times faster than the previous v0.23 release in single-threaded tests on an Apple M4 Max, according to benchmarks published September 21, 2026. The largest gain appears with the GPT-2 tokenizer, while T5-base records the low end of the range.
The release candidate keeps the same token IDs, vocabulary, merge ranks and public encoding interface as v0.23. That means applications can use the faster implementation without changing the token sequences sent to a model. Hugging Face says the benchmark covers ten model families and measures distinct documents rather than repeatedly encoding one cached document.
Thirty times faster without changing model inputs
Tokenization converts text into the integer IDs consumed by a language model. Hugging Face says that stage can become a bottleneck when teams train on very large datasets, serve many concurrent requests or repeatedly process long inputs. A slow tokenizer can leave an otherwise busy accelerator waiting for the CPU to prepare data.
Tokenizers v1 reaches 76% of linear scaling across eight workers in the company’s tests. The result comes from several changes to the encoding path rather than from a new tokenization format. The library continues to support multiple tokenizer families, including byte pair encoding, WordPiece and Unigram.
Measured result: Hugging Face reports a three-to-30-fold single-threaded encoding improvement over v0.23 on an Apple M4 Max, with identical output token IDs.
Replacing regex work and repeated allocation
Eight of the ten measured model families use byte pair encoding, or BPE. In the older path, a regular expression splits text into pre-tokens before the merge algorithm combines adjacent byte pairs according to their learned ranks.
Tokenizers v1 replaces that general-purpose regex work with a hand-written splitter called bitcannon for supported patterns. The splitter uses Boolean operations over bitstreams and SIMD instructions, allowing it to examine 64 bytes per register operation. Tokenizers whose patterns fall outside the supported set continue to use the regex path, so the reported gains vary by model.
The redesign also adds a thread-local word cache. When a pre-token appears again, the library can reuse its completed token IDs instead of running the merge process a second time. Hugging Face cautions that the cache helps most when inputs contain repeated pre-tokens and can add lookup work without much benefit on highly varied text.
A new merge loop built around reusable memory
The BPE merge loop now uses scratch memory supplied by the caller rather than allocating for each operation. It stores symbols in a flat array and links adjacent symbols by position, allowing a merge to update indices instead of moving data. Candidate pairs are packed into 64-bit values so the implementation can compare merge priorities as integers.
Batch processing reduces another source of overhead. Rather than calling the model stage once for every pre-token, v1 can process a batch of pre-token spans in one model call. Native parallelism gives each worker its own scratch buffer and word-cache area, removing the single lock that previously made concurrent encoding contend for shared state.
The release is still a candidate
Hugging Face has released the v1 candidate for Rust through crates.io. Developers can install it with cargo add tokenizers --pre; applications that only need encoding can disable the default training feature and install the encoding path without its C++ dependency.
The published measurements apply to the Rust crate. Hugging Face says its Python bindings use the same implementation but add per-call overhead that the benchmark does not include. The post also says additional model families will move to the new merge loop before version 1.0.0, with later work planned for the Transformers library and other projects built on Tokenizers.
The immediate change is narrower than a new model architecture but more practical for systems that process text at scale: the same input produces the same IDs, while the CPU spends less time splitting, allocating and merging it. The release candidate remains the version available for testing, and the company has not yet marked the 1.0.0 package as complete.
Read Hugging Face’s benchmark and implementation report.