刚刚，百度开源全新预训练基模，还是主打效率的Flash

Just now, Baidu open-sourced a brand-new pretraining model, still focusing on efficiency with Flash

The agent begins to execute tasks continuously, and the model’s requirements have changed.

As the task chain lengthens, the computational cost and long‑text capability of the model at each step affect the overall user experience and cost, making the importance of flash‑type models increasingly prominent: they focus on balancing capability, speed, and cost, handling high‑frequency and continuously running workloads.

On November 5, Baidu’s Wenxin series officially released the next‑generation foundation model Wenxin 5.5 Flash Base (ERNIE 5.5 Flash Base), and opened the weights.

This release is a sparse MoE language model with a total of 280B parameters, activating 14B per token, pretrained from scratch on 33.5T tokens, with a maximum training context of 128K, and during fine‑tuning and inference the context is extended to a long context of 1M tokens.

The model’s weights have been released, and it is supported by the open‑source ecosystem together with communities such as vLLM and SGLang.

• HF FP8: https://huggingface.co/baidu/ERNIE-5.5-Flash-Base

• HF BF16: https://huggingface.co/baidu/ERNIE-5.5-Flash-Base-BF16

• SGLang’s cookbook: https://docs.sglang.io/cookbook/autoregressive/Baidu/ERNIE-5.5-Flash

• vllm’s cookbook: https://recipes.vllm.ai/baidu/ERNIE-5.5-Flash

This release of Wenxin is a “Base” model that already possesses complete general capabilities and can serve as the starting point for all subsequent training. Users can perform domain adaptation, instruction fine‑tuning, reinforcement learning, and even behavior shaping for agents on this foundation. This means developers can decide the next direction of the model themselves.

What draws even more attention is the model’s performance and architecture. This time, Baidu delivers stronger foundational capabilities within limited activation parameters and cache budget.

SOTA performance at the same level

Currently, AI applications are about to enter the “Super Agent” era. With token consumption expanding dramatically and agents needing high‑frequency, long‑duration autonomous operation, flash‑level models will become the truly widely used tier. For the models themselves, agents need to frequently read massive environmental events, and the biggest bottleneck lies in long‑context pre‑fill (prefix pre‑fill) and KV cache memory usage; optimizing this is also the main thread throughout the Wenxin 5.5 Flash Base design.

Baidu’s goal is to make every activation parameter and each byte of KV cache contribute more. Based on this idea, its team has made extensive technical innovations in specific components such as attention, MoE, and future prediction.

The model has a total of 43 Transformer layers, a hidden size of 4096, and 64 attention heads. Except for the first layer, the feed‑forward networks in each layer are replaced with MoE blocks. Its pre‑training uses the Muon optimizer with FP8 mixed precision, and the context length is gradually expanded to 128K during training, then further extended to 1M during inference and fine‑tuning.

In the official same‑level model comparison, Wenxin 5.5 Flash Base leads overall on benchmarks such as knowledge, reasoning, mathematics, code, multilingual and long‑context, with competitors including DeepSeek-V4-Flash Base, Hy3-preview-Base and MiMo-V2.5-Base.

Compared item by item with DeepSeek-V4-Flash-Base (284B parameters, 13B activation), the new Wenxin model shows a clear advantage in subjects and reasoning, and its code capability is the most prominent strength: HumanEval+ scores 82.3 versus 69.5, LiveCodeBench 40.1 versus 31.9, FullStackBench 54.6 versus 43.3. However, Wenxin 5.5 Flash Base lags slightly on a few evaluations such as ChineseSimpleQA.

The results of long‑context evaluation also confirm the architectural efficiency; Wenxin 5.5 Flash Base is overall robust across six long‑context benchmarks, and the “few full layers plus many compressed layers” design does not sacrifice the model’s ability to handle long texts.

Behind this report card lies the Wenxin team’s extensive improvements and innovations to the model architecture.

Hybrid sparse attention, 1/7 full context access

In terms of architectural efficiency, Wenxin 5.5 Flash Base uses hybrid sparse attention to reduce inference cost.

When processing long texts, a Transformer must store KV cache for all past tokens at every layer, and the overhead grows linearly with context length, which is the main source of inference cost. Baidu’s insight is that not every layer needs to see historical details equally clearly.

Wenxin 5.5 Flash Base uses a hybrid MLA and HCA scheme: In every seven layers, only one layer retains token‑level access to the full context, using Multi‑Head Latent Attention (MLA), while the other six layers use Heavy Compressed Attention (HCA). Every 128 tokens are compressed into one cache entry, and attention is computed only on the compressed entries plus a recent short uncompressed window. Of the model’s 43 layers, 37 are HCA and only 6 are MLA.

During long-context inference, the MLA layer is responsible for page-by-page close reading, retaining access to each original token, while the HCA layer only focuses on chapter summaries, maintaining a coarse-grained view of the entire history and backtracking through a few layers only when details are needed. Because the HCA layer requires only a small portion of the full-context layer’s KV cache, the overall model’s caching overhead is greatly reduced.

The six-layer MLA is the only pathway to access long‑range history token by token. If dense attention is applied in these layers, they will consume most of the computation for long texts. Baidu therefore introduces sparse attention (DSA) in the MLA layers, using a lightweight indexer to score historical positions for each query, so that the core attention is computed only on the top‑k positions with the highest scores. The per‑query cost no longer grows with context length, while the pre‑training loss remains on par with dense attention.

When the KV cache budget is limited, the existing CSA + HCA approach heavily uses Compressed Sparse Attention (CSA), first compressing history by 4× and then sparsely selecting. Baidu on Ernie 5.5 Flash Base concentrates the budget on a few MLA layers that retain all tokens. Pre‑training comparisons with comparable budgets show that HCA combined with MLA outperforms pairing with CSA, sufficient to support fine‑grained retrieval.

Within the attention layer, Wenxin 5.5 Flash Base also includes two improvements. Low‑rank gated attention now uses the low‑rank query latent already generated inside the layer to compute the output gate; because the latent dimension is only a quarter of the hidden state, the gate projection parameters are reduced by 75%, and the ablation performance actually improves. Additionally, Virtual Head Attention (VHA) employs a lightweight cross‑head low‑rank mixture with identity skip connections to enhance information fusion, which can be merged into the operator during inference with negligible overhead.

The VHA-related paper “Virtual Head Attention” has been accepted to NeurIPS 2026.

Multi-router MoE, more accurate selection at the same cost

Attention determines how each token reads the history, and which component processes it after reading is the responsibility of the MoE layer.

The MoE (Mixture of Experts) architecture aims to achieve high intelligence density, and Wenxin 5.5 Flash Base provides a deep design for MoE layers, solving the problem of making arbitrary selections.

The traditional MoE workflow sends each token into the MoE layer, uses a single router to score it, and then selects experts via a hard top‑k. The problem with this mechanism is that the scores are only weakly correlated with the experts’ actual usefulness, especially in middle and deep layers. Moreover, because unselected experts do not receive gradients for that token, a wrong selection is hard to correct and the error reinforces itself.

Multi-Router allows multiple independently initialized sigmoid routers to score simultaneously, turning “one router decides” into “multiple routers evaluate collectively.”These routers have different preferences; an expert can be selected if it is strongly favored by any single router or by several routers at once, so a single misjudgment no longer determines the outcome. The number of activated experts, FLOPs, and communication overhead remain exactly the same as with a single router, incurring no extra cost during inference. According to the authors, the correlation between routing scores and expert utility improves significantly, with the weakest middle‑deep layers showing the greatest improvement.

In terms of configuration, each MoE block contains 512 routing experts, selects 10 at a time, plus one shared expert, and uses bias correction without auxiliary loss for load balancing; tokens are first projected from a 4096‑dimensional space to a 2048‑dimensional latent space, experts compute in the latent space and then map back to the original space, reducing communication and expert FLOPs by about half. The Stable LatentMoE design (RMSNorm combined with SiTU‑GLU activation) keeps expert outputs at a stable scale across 43 layers, and each expert’s routing weight is also multiplied by its own learnable scalar.

ECHO, even skips the MTP module

The third innovation of Wenxin 5.5 Flash Base takes a different direction, pointing to a more fundamental issue: what objective the model should use for learning.

The multi-token prediction (MTP) concept was systematically introduced by Meta in 2024 and has been widely adopted by the industry after being used in DeepSeek V3. It enables the model to learn to predict multiple subsequent tokens during training, providing a denser supervision signal to train the model’s planning ability. It can also serve as a draft model for speculative decoding during inference, guessing several tokens at once and then batch‑accepting them after verification to accelerate generation.

But the problem with conventional MTP is that it attaches a dedicated module outside the backbone Transformer to predict farther tokens, and the multi‑step loss mainly supervises these attachments, while each layer of the backbone still only serves the next token. In other words, the ability for long‑term planning is not directly trained.

ECHO does not add a separate MTP Transformer layer, but instead lets the backbone’s own final layer also handle future token prediction, so the multi‑step loss directly influences the parameters that generate the current token.Having these parameters predict farther tokens forces the backbone to plan the subsequent expansion of the sequence in advance. The model completes pre‑training with single‑step ECHO and achieves significant gains on math, reasoning, and coding tasks.

This design also brings an advantage in the inference stage: the final layer of the backbone that handles future predictions can directly serve as a built‑in draft model for speculative decoding, with almost no additional parameters. Notably, this layer also exhibits emergent extrapolation ability: during pre‑training it only learned to predict one extra token, but as a draft model it can continuously draft multiple tokens ahead, achieving a higher acceptance rate than an external MTP module trained in the same way. 

The three main improvements show that Baidu’s focus this time is forward‑looking; as agents begin to operate autonomously at high frequency and for long durations, the decisive factor in model competition is shifting toward efficiency, which is precisely the main theme that Wenxin 5.5 Flash Base set from the start.

More structural improvements, data, and infrastructure

There are also several component-level updates. In the residual connections, manifold‑constrained hyperconnection (mHC) replaces the standard residual structure; each layer reads and writes four parallel residual streams through a constrained mixing matrix, expanding inter‑layer residual bandwidth at near‑zero cost, aligning with the industry’s recent hyperconnection ideas.

In the output head, MultiMax introduces learnable piecewise temperatures to address the issue that a single softmax temperature can only choose between sparsity and multimodality. This suppresses the long tail of unreasonable tokens while preserving the diversity of reasonable candidates, with negligible training overhead and no extra memory usage.

For tokenization, a brand‑new BPE vocabulary of about 200 K tokens is used. Numbers are split from right to left in groups of three digits, so 1,234,567 becomes 1, 234, 567, aligning digit positions and making the model more friendly to arithmetic and digit‑dense texts.

On the data side, Baidu focuses on building three directions. Web data combines multilingual crawling, high-fidelity extraction, and model-driven quality filtering; code data, beyond static code, supplements software evolution data such as commits, issues, pull requests, code reviews, automated tests, and CI‑verified fixes, pointing to repository‑level reasoning and Agent software engineering, aligning with the Base positioning; long‑text data includes natural long documents, long sequences formed by concatenating distracting texts, as well as synthetic retrieval and multi‑hop QA data.

On the infrastructure side, Baidu keeps the distributed Muon optimizer efficient at the scale of thousands of GPUs. Muon uses per-tensor orthogonal updates, changing the communication pattern. Baidu switches Dense parameters from cross-machine large-ring communication to same-card paths, and merges MoE communication into a single collective operation, while evenly partitioning optimizer states across nodes to avoid excessive memory pressure on a few nodes.

Additionally, Baidu introduced the FlashMask-4 operator for large head dimensions, implemented with CuTe-DSL to support SM90 and SM100 architectures, capable of handling query-key heads of 576 dimensions and value heads of 512 dimensions. On SM100, its forward performance improves up to 175% and backward up to 74%. This operator has been open‑sourced in PaddlePaddle’s GitHub repository.

The infrastructure capabilities behind these models are the result of Baidu’s years of continuous investment in deep learning frameworks and large‑scale training.

In summary, on Wenxin 5.5 Flash Base, Baidu validated the gains of numerous technical improvements under the same architecture with a single complete large‑scale pre‑training. Real‑world deployment will be the next step to test this paradigm.

Base is just the starting point

Releasing a Base model also means giving developers more room for future development; in a sense, opening the Base requires more confidence than releasing a Chat version. The weights at the end of pre‑training directly showcase the model’s capabilities. Being willing to hand this checkpoint to the community for evaluation reflects confidence in the pre‑training ability.

Based on an open Base weight, the foundational model can be incorporated into developers’ secondary training pipelines, fostering a richer ecosystem. A high‑quality, reusable foundation means enterprise developers can train models for local languages, cultures, and industry scenarios without starting from scratch.

On the Wenxin 5.5 Flash Base model, from long‑text attention reading, to MoE routing expert invocation, and then ECHO attempting to let the backbone learn to predict the future more directly, designs centered on practical use also support more valuable tasks. In the future, with the same computational resources, AI will have the opportunity to support longer task chains, more rounds of attempts and feedback, thereby enabling unprecedented application modes.

Wenxin 5.5 Flash stable version is expected to be released soon. On this track with many players, opening the weights is just a ticket to entry; whether you can hold a steady seat at the table depends on whether the stable version can deliver on the base’s potential. For users, the arrival of Wenxin means another option, and competition in the Flash tier will become even fiercer, with cheaper and more powerful models.

Next, we just wait for the stable version to appear.

© THE END 

For reprints, please contact this public account for permission

Submit or request coverage: liyazhou@jiqizhixin.com