IFM Releases Six Apache 2.0 K2 Horizon Models

The Institute of Foundation Models (IFM), a frontier research lab established by MBZUAI in May 2025, has released K2 Horizon, a comprehensive suite of six open-source large language models. This release, described by IFM as the largest fully open-source model launch in AI history, includes models with parameter counts of 375 billion (375B-A23B), 36 billion (36B-A4B), 32 billion, 7 billion, 3.7 billion, and 0.9 billion. Alongside the models, IFM has made available the pre-training corpus, intermediate checkpoints, training code, configuration files, and detailed logs, fostering unprecedented transparency and accessibility for the AI research community. All six models are accessible on Hugging Face and are licensed under Apache 2.0, with support for both FP8 and GGUF formats, enabling broad deployment across various hardware platforms. Day-zero support is provided for popular inference engines such as vLLM, SGLang, and Ollama, and the models are compatible with NVIDIA, AMD, and Cerebras hardware. For users preferring hosted solutions, APIs are available through Compass, Cerebras, and Nebius via the platform.ifm.ai portal.

The K2 Horizon models share a unified core architecture, vocabulary (with the exception of the 0.9B model which uses a smaller vocabulary), training methodology, standardized interfaces, and deployment tooling. This architectural consistency is a key design principle, allowing development teams to prototype on smaller models, such as the 3.7B version, and seamlessly scale their applications to the largest 375B-A23B model without requiring modifications to their existing serving infrastructure. Each model in the K2 Horizon fleet was pre-trained on approximately 20 trillion tokens. A significant portion of this pre-training data, nearly 17%, comprised problem-solving trajectories that included explicit reasoning steps, aiming to enhance the models' logical capabilities. Approximately 10 trillion tokens of the training data were synthetically generated. IFM adopted a strategy of integrating post-training data throughout the mid-training phase rather than accumulating it solely at the end, a method that research teams report led to over 100 million unique synthesized tasks being incorporated. During training, tool definitions were presented in multiple formats, including JSON, XML, and Markdown, to ensure the models learned the underlying semantics of tools rather than just their syntactic structure. IFM found that Markdown proved to be the most efficient format for inference, achieving approximately 18.5% greater token efficiency on IFM's internal datasets compared to JSON.

A notable innovation within the K2 Horizon architecture is the Mixture-of-Value Attention (MoVA) mechanism. While conventional Mixture-of-Experts (MoE) models apply sparsity to their feed-forward layers, MoVA extends this expert routing principle to the multi-head attention mechanism itself. This introduces a second dimension for scaling model capacity by selectively activating attention heads. MoVA is designed to remain compatible with existing advanced attention techniques, including FlashAttention, grouped-query attention, and sparse attention, ensuring it can leverage and enhance current state-of-the-art performance optimizations. The K2-Horizon-MoVA-36B-A4B model, for instance, features 36 billion total parameters and incorporates this novel attention mechanism.