Soofi Consortium Releases Open Hybrid Mamba-Transformer MoE Model

A German research consortium has published the pretraining report for Soofi S 30B-A3B, an open base model designed for German and English language tasks. The model was trained end-to-end on Deutsche Telekom’s Industrial AI Cloud in Munich, with preview weights made available on Hugging Face. According to the report, Soofi S achieves the highest aggregate English and German scores among tested open base models.

Soofi S 30B-A3B is a Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model, comprising approximately 31.6 billion parameters with about 3.2 billion parameters activated per token. As a base model, it has not undergone instruction tuning, alignment, or safety tuning. The consortium coordinating this effort is the KI Bundesverband, with funding provided by the German Federal Ministry for Economic Affairs and Energy. Key participants in the consortium include Fraunhofer IAIS, DFKI, TU Darmstadt, ellamind, and Merantix Momentum.

The model's architecture features a stack of 52 layers, consisting of 23 Mamba-2 sequence-mixing layers, 23 granular MoE layers, and 6 Grouped-Query Attention (GQA) layers. Notably, only the 6 GQA layers maintain a KV cache. Each MoE layer is configured with 128 routed experts, activating 6 per token, and includes 2 shared experts. Additional architectural details include a model dimension of 2688, squared ReLU activation, RMSNorm, and the absence of positional embeddings.

The research team adopted the Nemotron 3 Nano reference design without modification, citing reasons of deployability on stacks like vLLM, serving efficiency, and the ability to maintain scientific control by fixing the backbone. This choice positions Nemotron 3 Nano as an architecture-identical baseline, allowing the data recipe to be the primary variable under investigation.