NeuroVFM Foundation Model Learns From Clinical Brain Scans

A University of Michigan research team has developed NeuroVFM, a novel neuroimaging foundation model trained on a massive dataset of clinical MRI and CT volumes. Published in Nature Medicine, NeuroVFM addresses the underperformance of general AI models on brain-imaging tasks by learning from uncurated clinical data. This approach, termed 'health system learning,' utilizes data generated during routine clinical operations, bypassing the need for manual radiology reports or disease-specific curation.

The NeuroVFM model was trained on 5.24 million clinical MRI and CT volumes, sourced from 566,915 studies within the UM-NeuroImages dataset. This data represents over two decades of patient care at Michigan Medicine. The core of NeuroVFM is its base model, Vol-JEPA, which extends previous I-JEPA and V-JEPA methods to handle volumetric medical images, aligning with a growing trend of JEPA-style learning in medical imaging.

Vol-JEPA operates as a self-supervised, vision-only algorithm. Instead of pixel reconstruction, it predicts representations within a learned latent space, eliminating the requirement for labels, report text, or voxel decoders. The training process involves tokenizing 3D volumes into patches, then using a student encoder to process visible context patches. A predictor then forecasts the latent representations of masked target regions, guided by a teacher encoder that provides ground-truth latents. The training objective minimizes the L1 loss between predicted and teacher latents, with gradients not backpropagated through the teacher model.