NVIDIA Releases Nemotron 3 Diarization Model

NVIDIA released Nemotron 3 Diarization, an open-weight speaker diarization model available on Hugging Face, designed to identify "who spoke when" in conversations. This 100 million-parameter model can track up to 8 distinct speakers simultaneously, even when their speech overlaps. A single checkpoint of Nemotron 3 Diarization supports both offline audio recordings and real-time streaming applications. The model's weights are distributed under the OpenMDW License 1.1, which explicitly permits commercial use. Deployment requires a Linux environment and NVIDIA NeMo software, leveraging GPUs from NVIDIA's Ampere, Ada Lovelace, Hopper, or Blackwell architectures. Speaker diarization is a critical component for many audio processing tasks, as automatic speech recognition (ASR) alone only transcribes words without attributing them to specific speakers. Without this attribution, understanding commitments, objections, or the flow of multi-party discussions becomes challenging. Diarization provides the time intervals during which each speaker is active, enabling the creation of speaker-attributed transcripts when combined with ASR output. This capability is fundamental for tools used in meetings, call analytics platforms, podcast production pipelines, and for maintaining memory in voice agents. Nemotron 3 Diarization represents an advancement over NVIDIA's prior Streaming Sortformer checkpoint, diar_streaming_sortformer_4spk-v2.1, which was limited to tracking 4 speakers. The new model effectively doubles this capacity to 8 speakers, addressing the need for accurate attribution in complex, multi-party audio scenarios where simultaneous speech is common. The architecture of Nemotron 3 Diarization processes 16 kHz, single-channel audio input in various formats including .wav, .flac, .opus, and .mp3. It first converts the audio into Mel-spectrogram features with a 10-millisecond step. These features are then stacked by a factor of 8, creating 80-millisecond encoder frames. A 31-layer Transformer encoder, incorporating rotary positional embeddings (RoPE), processes these frames. Following this, a Conv1D layer upsamples the predictions back to a 10-millisecond resolution. The model's output is a tensor of shape [T, 8], representing per-speaker activity probabilities for each time step. This design inherently handles overlapping speech, allowing multiple speaker channels to activate within the same frame if two or more individuals speak concurrently. Following the Sortformer methodology, the model orders speakers based on their arrival time in the conversation.