Moonshot AI Open-Sources MoonEP for Expert Parallelism

Moonshot AI has open-sourced MoonEP, an Expert Parallelism (EP) communication library specifically engineered for distributed Mixture-of-Experts (MoE) workloads. The release, announced during Kimi K3 Open Day, aims to make expert-parallel communication more efficient at scale and is distributed under an MIT license. MoonEP was introduced alongside other infrastructure codebases, including FlashKDA, which had previously been open-sourced, and AgentEnv, both published with this release. This new library is a key innovation credited with achieving a claimed 2.5× improvement in scaling efficiency for Kimi K3, a significant MoE model boasting 2.8 trillion parameters, native vision capabilities, and an extensive 1 million token context window.

The challenge MoonEP addresses stems from the inherent imbalance in expert parallelism. In this architecture, a router directs each token to its top-K experts, which are often distributed across different computational ranks. Routers frequently exhibit imbalance, leading to some experts receiving substantially more tokens than others. This skew is quantified by a metric called maxvio, defined as max_e (T_e / T̄) − 1, where T_e represents the tokens routed to expert 'e' and T̄ is the expected token count under a perfectly balanced scenario. A maxvio of 0 signifies perfect balance. The costs associated with this imbalance are structural, meaning they are fundamental to the design rather than incidental. The latency of a collective operation is dictated by its slowest participant, thus the performance of the hottest rank determines the iteration time. Compounding this issue, the token counts per rank fluctuate dynamically with each step. These dynamic activation shapes lead to GPU memory fragmentation and necessitate per-layer host synchronization, impacting overall training efficiency.

MoonEP's core innovation is its implementation of a hard invariant: every rank consistently receives exactly S × K tokens, regardless of the routing skew. Here, S denotes the input tokens per rank, and K represents the number of top-k experts routed per token. The library achieves this by dynamically planning a small number of redundant experts online, directly derived from the current router outputs. These duplicated experts are prefetched before the expert computation phase begins. During the backward pass, the gradients computed for these redundant experts are efficiently reduced back to their original home ranks. This design is structured around three key properties, ensuring perfect balance, efficient communication, and optimized resource utilization during the training of large-scale MoE models.