Nunchux AI Unveils VC-Attention for Faster Video Diffusion Transformers

Nunchux AI has introduced VC-Attention, a novel training-free low-bit attention kernel specifically engineered for video Diffusion Transformers (DiTs). This new kernel aims to simultaneously resolve two critical bottlenecks: value quantization error and the computationally intensive softmax stage within the attention mechanism. Video DiTs operate by flattening video clips into a sequence of spatiotemporal tokens and applying full self-attention at each layer. For context, a 5-second 720p video clip can generate approximately 70,000 tokens. On hardware like the RTX 5090, the attention computation consumes over 64% of the total generation time. Research from the Nunchux AI team indicates that attention accounts for roughly two-thirds of each denoising step in models like MiniMax-H3 when run on a single B200 GPU. While low-bit Tensor Cores can accelerate the two primary matrix multiplications (QK and PV) within attention, two significant obstacles persist. The first is the value term's contribution to output error, which prior methods like SageAttention2 failed to fully mitigate. These methods smooth queries and keys, but after this smoothing and rotation, the value term alone is responsible for 82% of the output error on the Wan2.2 benchmark. The second obstacle is the softmax operation, which still executes in FP32 precision. On high-performance GPUs such as the B200 and H200, this FP32 exponential calculation and its subsequent FP8 cast represent the longest stage in the processing pipeline. To address the issue of value outliers, Nunchux AI developed a technique called V-Smooth. Value outliers are concentrated in a small number of tokens and their channel distributions can vary across different attention heads, layers, and denoising steps. Traditional methods like Hadamard rotation, which preserve token norms, do not effectively remove these outliers, resulting in only a 0.2% change in value error. V-Smooth employs a different strategy involving three steps: Group, Demean, and Restore. In the Group step, an online k-means clustering algorithm groups value tokens within each batch and head. This permutation of keys and values together ensures that the non-causal attention output remains unchanged. The Demean step involves subtracting the mean from each 128-token hardware block. Only the residual values are then quantized, utilizing per-channel E4M3 at 8 bits or NVFP4 at 4 bits. Finally, the Restore step adds the mean back to the quantized residual using an online softmax that is already integrated into the row sum calculation, eliminating the need for a second pass or an additional buffer. Analysis across 100 Wan2.2 heads shows that the block mean effectively removes 8% of the block energy in sequence order, 12% under DeltaQuant's static cube, and a substantial 36% after sorting. Each mean calculation adds only 0.125 bits per value element. The grouping process is computationally efficient, running only during the initial 25% of denoising steps, and the resulting permutation is reused across four adjacent steps, further optimizing performance.