Perplexity and NVIDIA Unveil JOLT Reinforcement Learning Method

A key innovation of JOLT is that the teacher model is also trained, ensuring its feedback remains effective. This approach achieved accuracy comparable to GRPO on MATH but with 6.5 times fewer processed training tokens, representing a significant reduction in compute for reinforcement learning post-training.

JOLT was tested on various Qwen3 models (Qwen3-0.6B, Qwen3-4B-Instruct, Qwen3-8B, Qwen3-30B-A3B-Instruct) and DeepSeek-V2-Lite. The method utilizes one shared model for both teacher and student roles, eliminating the need for a separate teacher model.

Performance results show that JOLT+ achieved the highest mean score across all tested settings when compared against GRPO, OPSD, and π-Distill. Specifically, JOLT+ reached an AppWorld state-test pass rate of 72.09%, a 9.9-point improvement over GRPO's 62.20%.

The research builds upon On-Policy Self-Distillation (OPSD) and addresses limitations in outcome-reward RL methods like GRPO, which provide sparse rewards that do not indicate which tokens contributed to success. JOLT aims to fix this sparsity by having a stronger teacher score each student token.

The researchers argue that a teacher with access to the correct answer might take shortcuts unavailable to the student, potentially limiting guidance. They observed that OPSD's teacher-student disagreement grew during MATH training, suggesting a need for JOLT's simultaneous training of both roles.