NVIDIA AI Releases Molt RL Framework

NVIDIA's NeMo team has released Molt, a new PyTorch-native framework for agentic reinforcement learning, designed to streamline the research process by reducing the complexity of algorithm modification. The framework aims to be compact enough for researchers to understand its entirety and for AI coding assistants to process. NVIDIA states that Molt's codebase comprises approximately 8.6K lines of reinforcement learning code, a figure derived by tracing the import graph from each framework's RL entry point. This contrasts with other frameworks, where verl is measured at about 62K lines, slime at 25K lines, and OpenRLHF at 7.2K lines. Molt is available under the Apache 2.0 license and includes launch codes, Slurm scripts, and a prebuilt container, indicating its deployability. However, NVIDIA positions Molt primarily as research infrastructure rather than a production training service, with hardware requirements being a significant factor for its use. The provided recipes for Molt assume access to two nodes, each equipped with eight NVIDIA H100 GPUs, with a split of eight GPUs for training and eight for rollout. This hardware specification places Molt within reach of advanced research institutions and well-funded organizations, including frontier and adjacent labs, substantial AI startups engaged in post-training activities, enterprise AI research groups in sectors like finance, healthcare, and robotics that train agents within proprietary environments, and academic laboratories possessing multi-node H100/H200 GPU clusters. The framework supports a range of applications, such as multi-turn tool-use agents, code-execution agents, vision-language environments (exemplified by the shipped geo3k recipe), reward loops utilizing LLMs as judges, and the distillation of on-policy learning onto smaller student models. Molt's architecture is built around three core components orchestrated within a single loop: Ray for managing placement and asynchronous queues, vLLM for handling rollouts, and NVIDIA AutoModel with FSDP2 for the training process. A key design choice is the utilization of existing, un-forked components, ensuring that upstream improvements to Ray, vLLM, and NVIDIA AutoModel are integrated into Molt through container updates rather than requiring extensive code rebasing. The runtime environment consists of an agent pool, which comprises a set of vLLM engines managed by a request router, and a singular trainable policy actor. This setup facilitates efficient agent execution and policy updates within the reinforcement learning loop. The framework's design prioritizes ease of modification and comprehension for researchers, aiming to accelerate the pace of innovation in agentic reinforcement learning by abstracting away much of the underlying infrastructure complexity.