Moonshot AI Releases Kimi K3: 2.8T Open MoE Model

Moonshot AI released Kimi K3 this week, a 2.8-trillion-parameter open Mixture-of-Experts (MoE) model that includes native vision capabilities and a 1-million-token context window. The company claims Kimi K3 is the world's first open model to reach the 3-trillion-parameter class. This new model is built upon two key architectural innovations: Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), which are designed to optimize information flow across sequence length and model depth.

Kimi K3 is specifically engineered for tasks requiring long-horizon coding, extensive knowledge work, and complex reasoning. Moonshot AI stated that while Kimi K3's overall performance trails leading proprietary models such as Claude Fable 5 and GPT 5.6 Sol, it consistently outperformed other tested models on Moonshot's proprietary evaluation suite. The company has been a consistent leader in open-model sizes, with Kimi models setting the upper bound for the past nine out of twelve months.

The Kimi Delta Attention (KDA) mechanism is a hybrid linear attention approach that Moonshot AI reports enables decoding speeds up to 6.3 times faster within million-token contexts. Attention Residuals (AttnRes) operate along the depth axis, selectively retrieving representations rather than accumulating them uniformly, which Moonshot AI claims results in approximately 25% higher training efficiency at a minimal additional cost of under 2%. Sparsity is another critical component, with K3 utilizing Stable LatentMoE, activating 16 out of 896 available experts.

To manage the challenges of routing and optimization at this sparsity level, Kimi K3 employs Quantile Balancing for expert allocation, derived directly from router-score quantiles, which removes the need for heuristic updates and sensitive hyperparameter tuning. Further enhancements include Per-Head Muon for independent optimization of attention heads and Sigmoid Tanh Unit (SiTU) and Gated MLA for improved activation control and attention selectivity. These architectural changes, combined with refined training and data strategies, contribute to an estimated 2.5 times better overall scaling.