The Pulse

Baidu Releases ERNIE 5.5 Flash Base Weights

Baidu released ERNIE-5.5-Flash-Base, a model with 280B total parameters and 14B activated per token. The pre-training checkpoint is available in FP8 and BF16 formats for further training and inference.

AI.info Team · Written with AI · fact-checked · 2026-11-06

Baidu publishes the pre-training checkpoint

Baidu released ERNIE-5.5-Flash-Base on November 5, 2026, as a set of base weights intended for further training. The sparse Mixture-of-Experts model has 280B total parameters, with 14B activated per token. Its checkpoint comes from the end of pre-training, before any mid-training or post-training.

The release presents the weights as a starting point for domain adaptation, instruction tuning and reinforcement-learning research. Releasing the weights, Baidu says, lets developers build on the general capabilities already in place and choose for themselves where to take the model next. The company also describes the checkpoint as a clean research baseline for comparing training methods under controlled conditions.

Hybrid attention compresses distant context in most layers

ERNIE 5.5 Flash uses 43 Transformer layers, a hidden size of 4096 and 64 attention heads. Its hybrid attention design gives token-level access to the full context in one layer out of seven; the other layers use Heavily Compressed Attention, which stores one cached entry for every 128 tokens. Those layers also attend to a short window of recent, uncompressed tokens.

The model’s MoE blocks use 512 routed experts, with 10 selected per token, alongside one shared expert. All layers except the first use an MoE feed-forward block, and the experts operate in a 2048-dimensional latent space. ERNIE 5.5 Flash also replaces the standard residual connection with four parallel residual streams connected through a constrained mixing matrix.

Training data targets code and long documents

Baidu pre-trained the model from scratch on 33.5T tokens using the Muon optimizer and FP8 mixed precision. The training context expanded gradually to 128K tokens, while fine-tuning and inference support a 1M-token context. The released weights have not gone through mid-training or post-training.

The described corpus includes multilingual web material, code from repositories and the web, and long-context data. Code examples include commits, issues, pull requests, code reviews, generated tests and CI-verified fixes. For long-context training, the release describes natural long documents, sequences combined with retrieved distractors, and synthetic retrieval and question-answering data.

Benchmarks compare four released base models

Baidu evaluated ERNIE-5.5-Flash-Base against DeepSeek-V4-Flash-Base, Hy3-preview-Base and MiMo-V2.5-Base. The four publicly released checkpoints were tested with the same evaluation harness, prompts, few-shot examples and LLM judge for free-form question answering, the release says. The comparison covers 30 benchmarks across seven categories.

In the listed results, ERNIE 5.5 Flash Base scores 93.3 on MMLU, 60.2 on GPQA and 82.3 on HumanEval+. It records 91.7 on RepoQA and 48.7 on LongBench-v2.

FP8 and BF16 weights are available

The model is available on Hugging Face as FP8 and BF16 safetensors under the ERNIE 5.5 Flash Base Model License Agreement v1.0. SGLang and vLLM can serve the model through the completions API, and the release points users to a separate example for fine-tuning. The release also specifies that inputs should begin with a BOS token, with add_special_tokens=True retained during tokenization.