NVIDIA Compresses Nemotron-3 LLM for 2x Server Throughput

NVIDIA AI has released Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of its Nemotron-3-Super large language model, designed to improve serving efficiency. The original Nemotron-3-Super model features 120.7 billion total parameters with 12.8 billion active parameters. The new compressed version, Nemotron-Labs-3-Puzzle-75B-A9B, reduces the total parameters to 75.3 billion and active parameters to 9.3 billion, while preserving the hybrid Mamba-Transformer Mixture-of-Experts (MoE) architecture. This compression was achieved by reducing the Mamba SSM state size from 128 to 96 and the MoE routed expert intermediate size from 2688 to a range of 1280-2688, among other adjustments. The deployment target for this compressed model was set prior to the architecture search, aiming for a 2x increase in server throughput at 100 tokens per second per user, and the ability to handle 8 concurrent 1 million-token requests on a single H100 GPU. Testing indicates that Nemotron-Labs-3-Puzzle-75B-A9B achieves an 8xB200 total throughput increase of 1.60x to 2.14x compared to Nemotron-3-Super, when using matched NVFP4 precision and matched user throughput. Furthermore, the single-H100 1 million-token concurrency has increased from 1 to 8, attributed to a significant reduction in weight size from 70 GB to 44.5 GB. The research also highlights that an iterative approach to compression, termed "Iterative Puzzle," outperformed a single-step compression method by an average of 0.57 points on specific benchmarks, while maintaining the same compression target. Performance evaluations on benchmarks such as Arena-Hard-V2 and SWE-Bench show minor decreases of -4.2 and -2.6 points respectively, with minimal impact on RULER and AA-LCR benchmarks. The Nemotron-Labs-3-Puzzle-75B-A9B model retains the parent model's 88-block layout, comprising 40 Mamba blocks, 40 MoE blocks, and 8 attention blocks. The changes are concentrated within the capacity of these blocks, with the number of routed experts, shared expert size, and MoE latent size remaining unchanged. Attention layers were also left unmodified, as the research team noted that Nemotron-3-Super already demonstrates high KV-cache efficiency.