AMD Releases Open Instella-MoE-16B-A3B LLM

Advanced Micro Devices (AMD) has released Instella-MoE-16B-A3B, a fully open Mixture-of-Experts (MoE) large language model trained from scratch on its Instinct MI300X and MI325X Graphics Processing Units (GPUs). This model boasts a total of 16 billion parameters, but crucially, it activates only 2.8 billion parameters per token during inference, a characteristic of MoE architectures designed for efficiency. AMD is making the model's weights available from every training stage, alongside its data mixtures, training configurations, and inference code, fostering transparency and enabling further research.

The release incorporates two significant systems-level architectural choices: Gated Multi-head Latent Attention (MLA) and FarSkip-Collective connectivity. Gated MLA enhances the standard MLA by introducing a lightweight, learned output gate. This gate is derived from an input-conditioned linear projection and is applied multiplicatively before the final output projection, allowing for more dynamic control over attention outputs. FarSkip-Collective connectivity is designed to manage the flow of information, particularly for outdated or partial activations, which is vital for efficient large-scale model operation.

While the model's weights are released under a ResearchRAIL license, restricting their use to academic and research purposes, the training codebase is available under the permissive MIT license. This makes the codebase a more reusable asset for broader applications. AMD positions this release for AI research labs, university groups, and enterprise R&D teams possessing data-center GPU capacity, rather than lean startups seeking readily deployable commercial endpoints. Key industries that can benefit include semiconductor and cloud infrastructure providers, AI tooling vendors, and academic research institutions.

Potential applications for Instella-MoE-16B-A3B include reproducing end-to-end MoE training recipes, studying expert-parallel serving strategies, evaluating long-context behavior up to 128,896 tokens, and conducting post-training reinforcement learning experiments. The model's architecture is decoder-only, featuring 27 layers, a hidden size of 2048, 16 attention heads, and a vocabulary size of 128,896 tokens. Each MoE layer is configured with 2 shared experts and 6 routed experts selected from a pool of 64, contributing to the 2.8 billion active parameters. A Multi-Token Prediction objective was employed during its pre-training and mid-training phases. For serving, the 16 billion parameters in BF16 precision require approximately 32 GB of weight memory, meaning a single high-memory accelerator is sufficient. AMD is providing the SGLang inference code to facilitate deployment.