The Pulse
Hugging Face Adds Tencent’s 780B Hy4 to Transformers 5.17.0
Hugging Face Transformers 5.17.0 adds support for Tencent’s Hy4-Preview, a mixture-of-experts model with 780 billion total parameters, 49 billion active per token and a 1 million-token context window. The release also adds VibeVoice, NeoMME
AI.info Team ·
780 billion parameters, 49 billion active per token
Hugging Face’s Transformers 5.17.0 adds native support for Tencent’s Hy4-Preview, a mixture-of-experts language model with 780 billion total parameters. The model activates 49 billion parameters for each token, routes tokens through eight of 256 routed experts, and includes one shared expert that remains active across the model.
The release landed on September 9, 2026, and gives developers a standard Transformers path for loading the model rather than relying only on specialist inference runtimes. Hugging Face’s release notes describe a context window of 1 million tokens, although the full checkpoint cannot fit on a single accelerator and requires tensor or expert parallelism.
Arthur Zucker, a Transformers engineer at Hugging Face, described the wider fifth-generation release in a LinkedIn post as “the apex of my 3 years @hugginface: transformers v5!” He added: “To the entire AI community: we are ready to host every single model you can come up with.”
Zucker’s post predates version 5.17.0, but it captures the project’s stated aim: make the library broad enough to accommodate model designs that do not fit the original Transformer template.
Hy4 combines sparse attention with four residual streams
Hy4-Preview is not simply a larger conventional Transformer. The implementation combines Multi-head Latent Attention, DeepSeek Sparse Attention, gated attention with learnable sinks and Independent Hyper-Connections.
Multi-head Latent Attention compresses keys and values into a lower-dimensional latent representation before expanding them back for the query heads. DeepSeek Sparse Attention uses a lightweight indexer to select the keys each query should inspect. In layers marked as “shared,” Hy4 reuses the selection made by the previous full indexer layer instead of calculating a new index for every layer.
Independent Hyper-Connections replace a single residual path with four parallel residual streams. Those streams are combined before each sublayer and distributed again afterward, changing how information moves through the network without changing the model’s basic decoder-only interface.
The release notes also clarify a boundary in the initial integration: Transformers does not execute Hy4’s multi-token prediction layers. The released checkpoints retain those weights so other runtimes can use them for speculative decoding, but Transformers ignores them when loading the model.
Why the 780B figure differs from Tencent’s 770B specification
Tencent’s model card describes the Hy4 backbone as a 770 billion-parameter model. It contains 78 layers: the first uses a dense feed-forward block, while the remaining 77 use mixture-of-experts blocks with 256 routed experts and one shared expert. The card separately lists a native multi-token prediction layer with 10 billion parameters, producing the 780 billion total referenced by Hugging Face.
That distinction matters for deployment. The 770 billion figure describes the backbone used for ordinary forward passes, while the 780 billion figure includes the retained speculative-decoding component. The model card says each token activates eight routed experts alongside the shared expert, while the Transformers release summarizes the active count as 49 billion parameters.
Tencent released Hy4-Preview under the Apache 2.0 license on August 28, 2026, according to the company’s announcement. The model is available in a full version and an FP8 variant through Tencent’s model distribution channels, with deployment guidance for vLLM and SGLang as well as Transformers.
Transformers 5.17.0 adds six other model families
Hy4 is the largest addition in the release, but it is not the only new architecture. Version 5.17.0 also adds VibeVoice, a long-form speech synthesis framework that generates multi-speaker audio with a next-token diffusion process inside a language-model structure.
NeoMME brings multilingual multimodal encoders from H Company, including a 260 million-parameter model and an 800 million-parameter model that process text tokens and raw image patches in one bidirectional encoder. Its Retriever variant supports both late-interaction scoring and dense embeddings for visual document search.
Alibaba DAMO Academy’s Fun-ASR-Nano adds an 800 million-parameter speech-recognition model with Chinese, English and Japanese support, seven Chinese dialects, 26 regional accents, hotword customization and native punctuation. Moonshot AI’s Kimi Linear adds a hybrid architecture built around Kimi Delta Attention, while NVIDIA’s Canary-1B-v2 targets multilingual speech recognition and speech-to-text translation.
The release also adds NeuCodec, a neural audio codec based on finite scalar quantization. Hugging Face says it operates at 50 tokens per second and 0.8 kilobits per second when using 16 bits per token, with 16 kHz input and 24 kHz output.
Vision RoPE changes will affect custom integrations
Beyond new models, Transformers 5.17.0 introduces a breaking change for vision systems. Two-dimensional and three-dimensional rotary embeddings now use a unified frequency-computation module in modeling_rope_utils.py.
Developers with custom vision models that depend on attention-layer or model-specific RoPE grid interleaving must move that logic to the centralized implementation. The release also includes generation changes: it avoids unnecessary accelerator synchronization on every decode step, stops generation from unconditionally downloading remote Hub files and standardizes the past_key_values argument across AfMoE models.
For Hy4 users, the immediate question is less whether Transformers can recognize the architecture than how many accelerators are available to run it. Hugging Face’s own documentation recommends tensor parallelism or expert parallelism, and says expert parallelism is inference-only because the routed-expert all-reduce has no backward pass. The software support is now in the main library; the hardware bill still belongs to the deployment team.