Tencent Open-Sources AngelSpec for Speculative Decoding

Tencent has open-sourced AngelSpec, a torch-native training framework designed to enhance speculative decoding for large language models. This framework specifically addresses the challenges of workload heterogeneity by offering specialized training for both autoregressive multi-token prediction (MTP) and block-parallel speculative decoding, particularly for Hy3 models. Speculative decoding is a technique that aims to accelerate text generation by using a lightweight "drafter" model to propose multiple future tokens, which are then verified by a larger "target" model in a single forward pass. The efficiency of this process hinges on two factors: the acceptance rate of the proposed tokens and the computational cost of the draft-and-verify step. These factors often exhibit an inverse relationship across different types of tasks. In open-ended conversational scenarios, where many continuations are semantically plausible, the acceptance rate of draft tokens tends to decrease rapidly as the proposed sequence length increases. This makes generating and verifying long blocks computationally wasteful. For such tasks, AngelSpec employs an autoregressive MTP drafter that proposes shorter candidate sequences, aligning better with the characteristics of conversational data. Conversely, workloads involving code generation or mathematical reasoning exhibit stronger constraints on future tokens due to factors like programming syntax, formal expressions, and step-by-step derivations. These tasks often involve longer predictable spans, which are more effectively handled by block-parallel drafting methods. To cater to this diversity, AngelSpec provides two distinct drafters: an MTP model trained on conversational data and a block-diffusion model optimized with code and mathematics-focused samples. This dual approach allows AngelSpec to tailor the speculative decoding process to the specific demands of different workloads, moving beyond a one-size-fits-all universal drafter. The framework's design emphasizes treating workload heterogeneity as a fundamental constraint, influencing the structure, training data, and verification depth of the models. This contrasts with prior speculative decoding research, which often focused on a single drafter optimized for an averaged benchmark mixture. AngelSpec's commitment to lossless speculative decoding means that the acceleration achieved does not come at the cost of output accuracy. The original Hy3 model, which AngelSpec supports, is trained with a single MTP layer and without recurrent self-conditioned unrolling, with the block being reusable during inference. This open-source release by Tencent aims to provide researchers and developers with a more adaptable and efficient tool for improving the performance of large language models across a wider range of applications.