NVIDIA Compresses Nemotron-3 LLM, Boosting Server Throughput

NVIDIA AI has released Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of its Nemotron-3-Super large language model, designed to improve serving efficiency. The original Nemotron-3-Super model features 120.7 billion total parameters and 12.8 billion active parameters, utilizing a hybrid Mamba-Transformer Mixture-of-Experts (MoE) architecture. The newly released compressed model, Nemotron-Labs-3-Puzzle-75B-A9B, reduces the total parameter count to 75.3 billion and active parameters to 9.3 billion, while preserving the 88-block hybrid layout which includes Mamba, MoE, and attention blocks.

This compression strategy targets significant improvements in server throughput. NVIDIA AI aimed for a 2x increase in server throughput at 100 tokens per second per user and the ability to handle 8 concurrent 1 million token requests on a single H100 GPU. The compressed model achieves an 8xB200 total throughput increase of 1.60x to 2.14x compared to Nemotron-3-Super, when using matched NVFP4 precision and matched user throughput. Furthermore, single-H100 1 million token concurrency has been increased from 1 to 8, attributed to a reduction in weight size from 70 GB to 44.5 GB.

The compression involved reducing the Mamba SSM state size from 128 to 96 (75% capacity) and the MoE routed expert intermediate size from 2688 to a range of 1280-2688, averaging a 59.9% reduction. The number of activated routed experts per token also decreased from 22 to a range of 4-18, averaging a 50% reduction. The active routed expert capacity was reduced to a mean of 30.9%. Attention layers remained unchanged, as the research noted Nemotron-3-Super's existing KV-cache efficiency.

NVIDIA AI has made three checkpoints of Nemotron-Labs-3-Puzzle-75B-A9B available on Hugging Face, supporting BF16, FP8, and NVFP4 formats. Performance evaluations indicate that the iterative compression approach, termed "Iterative Puzzle," outperformed single-step compression by an average of 0.57 points on the same compression target. While benchmarks like Arena-Hard-V2 and SWE-Bench showed minor performance decreases of -4.2 and -2.6 respectively, metrics like RULER and AA-LCR remained largely unaffected.