Alibaba Qwen Team Releases 125B Multimodal MoE Model

Alibaba's Qwen team has released Qwen3.8-Flash-Next, an open-weight multimodal Mixture-of-Experts (MoE) model designed for cost efficiency per token. This new model features a 125 billion parameter backbone, augmented by a 51 billion parameter N-gram embedding table and a 4 billion parameter multi-token prediction module, resulting in a total of 180 billion parameters on disk. Crucially, only 6 billion parameters are activated per token, a characteristic of MoE architectures that significantly reduces computational requirements during inference. The release serves as an early preview of the architectural advancements that will be integrated into the upcoming Qwen4 model, mirroring the role Qwen3-Next played in the development of Qwen3.5.

Four key innovations distinguish Qwen3.8-Flash-Next. Firstly, it employs a hybrid attention mechanism combining Gated DeltaNet (GDN) and Qwen Sparse Attention (QSA). In this configuration, three out of every four layers utilize GDN, a linear attention layer designed to compress historical context into a fixed-size recurrent state. The remaining fourth layer incorporates QSA, which employs a lightweight indexer to selectively choose relevant context at a micro-block granularity rather than on a per-token basis. The layer arrangement follows a pattern of 12 repetitions of (3 x GDN layers followed by 1 x QSA layer) across the model's 48 layers, with the QSA mechanism allocated a budget of 512 blocks, equivalent to 2048 tokens. Secondly, the model incorporates Gated Residual connections, where the residual stream broadens into four parallel branches. These branches are managed by an element-wise read gate and per-branch scalar write gates, operating at a bottleneck rank of 320. Thirdly, an N-gram Embedding component, comprising a 20 million entry table for bigrams and trigrams, is integrated at layer 2. This module enhances the model's capacity through deterministic lookups and can be offloaded to host memory. Finally, the model utilizes the Muon optimizer, a new optimization algorithm developed by the Qwen team.

The Qwen team reports that the training cost for Qwen3.8-Flash-Next is approximately one-ninth of that for Qwen3.7-Plus, highlighting the efficiency gains achieved. While the model is deployable, it requires substantial hardware resources. The FP8 checkpoint measures 172.78 GiB, and the BF16 checkpoint is 335.28 GiB. For deployment using FP8 precision, the vLLM recipes recommend a Tensor Parallelism (TP) configuration of at least TP2 on GB300 GPUs, with TP4 being the preferred option. For an 8xH200 node, the TEP8 configuration is advised, as plain TP8 is incompatible with the checkpoint's 128-wide quantization blocks. It is important to note that the sparse activation implemented in this model reduces computational load but does not affect storage requirements.