NVIDIA Releases Audex Unified Audio-Text LLM

NVIDIA released Audex (Nemotron-Labs-Audex-30B-A3B) this week, a unified audio-text large language model capable of understanding and generating both audio and speech. A key design goal for Audex was to avoid the "text tax" often seen in multimodal models, where performance on text-based benchmarks degrades when audio or vision capabilities are added. NVIDIA's research team reported that this regression occurs even in speech-only output models. Audex, however, maintains the text intelligence of its backbone model, Nemotron-Cascade-2-30B-A3B, with text benchmark scores showing only minor gains or losses.

Audex is structured as a single 30 billion parameter Mixture-of-Experts (MoE) Transformer decoder, with 3 billion parameters activated per token. Its backbone, Nemotron-Cascade-2-30B-A3B, is a text-only MoE LLM featuring a hybrid Mamba-Transformer architecture with 52 layers, 128 routable experts, and 6 activated experts. The model's unified design simplifies audio processing by encoding audio inputs and projecting them into the text embedding space. During generation, both text tokens and quantized audio tokens are treated uniformly, eliminating the need for separate "thinker-talker" components or cascaded model structures. This streamlined approach allows Audex to run on standard LLM infrastructure, including Megatron-LM for training and vLLM for inference.

The model incorporates an audio encoder, AF-Whisper from Audio Flamingo 3, which is based on the Whisper Large-v3 architecture and supports 16kHz input. Two-layer MLP adapters are used to map audio features into the model's dimension. Audex supports both an instruct mode and a thinking mode, with a context length that can reach up to 1 million tokens. Checkpoints for Audex, including a smaller Audex-2B variant, have been released under a noncommercial license. The development team employed a multi-stage Supervised Fine-Tuning (SFT) process combined with text-only Cascade Reinforcement Learning (RL) to achieve this unified audio-text capability without sacrificing text performance.