The Pulse
Transformers 5.21 Adds MiniCPM-OCR and Flint MoE Support
Hugging Face released Transformers 5.21.0 on October 26, adding support for OpenBMB’s MiniCPM-OCR and Recursive’s Flint mixture-of-experts model. The update also changes tensor- and expert-parallel plans, so some custom configurations need
AI.info Team · Written with AI · fact-checked ·
Transformers 5.21.0 adds native support for MiniCPM-OCR and Flint, but the upgrade also changes parallelism rules that can force maintainers to revise custom configurations. Hugging Face published the release on October 26, pairing model integrations with changes to how tensor parallelism handles embeddings and how mixture-of-experts models shard their experts.
The additions target different workloads: MiniCPM-OCR reads text-dense images, while Flint brings Recursive’s mixture-of-experts language model into the library. For teams already using custom tensor- or expert-parallel plans, the release notes flag migration work alongside the new model support.
MiniCPM-OCR targets text-heavy images
MiniCPM-OCR belongs to OpenBMB’s MiniCPM-V family and retains the MiniCPM-V 4.6 architecture, according to the release notes. That design combines a SigLIP vision encoder and a window-attention merger with a Qwen3.5 language model. The model is specialized for reading images containing dense text, including scanned pages, screenshots, receipts and tables.
The release adds the model to Transformers rather than presenting a new benchmark or a new model launch. Its practical change is that developers can use the library’s native model support for these document-understanding tasks. Hugging Face lists documentation and a paper alongside the integration.
Flint routes tokens through a sparse expert system
Flint is Recursive’s mixture-of-experts language model. Its attention design combines sliding-window attention with full attention: most layers use rotary embeddings over the current token and the previous 512 tokens, while the remaining layers use full attention without positional encoding. The release notes also describe learned attention sinks and per-head RMS-normalized queries and keys.
In each sparse layer, Flint routes a token to six of 256 experts using sigmoid scores, then adds an ungated shared expert; its first two layers are dense. That routing setup is the model’s distinguishing implementation detail in this release, not a performance claim. Transformers adds model support through a contribution credited to Lovish Madaan.
Six additions, from document retrieval to speech
MiniCPM-OCR and Flint are two of six additions listed in the release. The others are UltraVBert for visual document understanding and retrieval, Gradium Phonon TTS, Pocket TTS, and the Pocket TTS audio tokenizer. The lineup spans image-based document work, language modeling and speech generation.
Pocket TTS is described as a 100-million-parameter text-to-speech model that runs faster than real time on a CPU. Gradium Phonon TTS is an on-device speech model family. The release also adds the Pocket TTS tokenizer, which converts audio to and from continuous latents.
Custom parallel plans need attention
For infrastructure teams, the release’s most consequential changes may be outside the new model integrations. Tensor parallelism now gives embeddings a dedicated style and shards input embeddings by default, including when embedding widths are not divisible by the parallel size. The release notes say custom plans that applied ColwiseParallel or RowwiseParallel to embeddings must switch to the new embedding style.
Every MoE tensor-parallel and expert-parallel plan must now shard its experts. Custom or downstream plans that leave experts replicated across ranks need updating to meet the new requirement. Transformers 5.21 also adds an FP8 backend for QuantizedCache, giving users another option for storing key-value states in a quantized cache.
The release therefore delivers both new model integrations and changes to existing distributed setups. Teams adopting Flint or other MoE models should check their parallel plans; teams with custom embedding plans need to replace the old linear-layer styles on embeddings.