Black Forest Labs Releases FLUX 3 Action For Robot Control

Black Forest Labs (BFL), a research entity known for its FLUX image models, has launched FLUX 3 Action, a 7-billion parameter open-weights World Action Model (WAM) designed for robot control applications. This model processes camera frames, robot state information, and textual instructions to predict future video frames and subsequent action sequences. On the RoboLab-120 benchmark, FLUX 3 Action achieved a task success rate of 42.92%, securing the top position. Deployment of FLUX 3 Action is feasible, though it has specific hardware requirements. The DROID policy necessitates approximately 32 GB of GPU memory when operating in BF16 precision on an H200 GPU. For systems with 24 GB graphics cards, it can be utilized with FP8 quantization and offloading of the text encoder. The FLUX Kommunity License permits non-commercial use of the model. Open robot policies often involve trade-offs between prediction accuracy and computational cost. For instance, NVIDIA's Cosmos 3 Nano, another WAM, leads RoboLab with a 36.8% success rate but is computationally intensive due to its video prediction capabilities. Conversely, Vision-Language Agents (VLAs) like π0.5 are faster but achieve lower success rates, around 28.0%. BFL's internal measurements on a B200 GPU indicated that Cosmos 3 Nano (in FP8) requires approximately 4.7 times more processing time per second of robot motion compared to π0.5 (in BF16). FLUX 3 Action aims to bridge this performance gap by jointly predicting video and actions, employing a smaller backbone architecture and distillation techniques to enhance efficiency. The architecture of FLUX 3 Action is an extension of the multimodal FLUX 3 backbone. Its pretraining phase incorporated image, video, and audio data, with video data constituting over 95% of the training tokens. The model encodes text, video, and robot state inputs into tokens. The backbone then decodes future tokens into predicted video frames and action tokens that translate into robot actions. During midtraining, the dataset was balanced between pretraining data (36.95% of samples) and action-aligned video data (63.05%). This action data was sourced from game recordings, egocentric human hand videos, data from handheld grippers, and teleoperation across 14 different embodiments. The majority of the robot data utilizes a standardized 50-dimensional end-effector action space, referred to as EE50. The significance of pretraining is highlighted by BFL's findings: without it, training a DROID-only model resulted in performance below 1% on RoboLab. However, with pretraining, the same training protocol yielded an 11.6% success rate. The RoboLab-120 benchmark comprises 120 distinct tabletop tasks simulated in Isaac Sim, with each task undergoing 10 trials on a Franka robot setup styled after DROID.