Xiaomi MiLM Plus Releases PROVE Object Removal Metrics and Benchmark

MiLM Plus, a research team within Xiaomi Inc., has introduced PROVE (Perceptual RemOVal cohErence), a novel system designed to address the limitations of existing metrics in evaluating object removal models. This new system, accepted for presentation at ACM MM 2026, comprises two perception-aligned metrics, RC-S for spatial coherence and RC-T for temporal consistency, alongside PROVE-Bench, a two-tier real-world video benchmark. Object removal models, particularly those employing diffusion erasure techniques, have advanced significantly, capable of convincingly reconstructing shadows, reflections, and occluded structures. However, traditional evaluation metrics such as PSNR, SSIM, LPIPS, ReMOVE, and CFD frequently fail to accurately rank the performance of these advanced models, often producing incorrect rankings. The fundamental issue lies in the ill-posed nature of object erasure, which is a one-to-many task, meaning there isn't a single, definitive ground truth against which to compare generated outputs. PROVE tackles this challenge by scoring the edited region locally, utilizing sliding-window Maximum Mean Discrepancy (MMD) applied to DINOv2 features. Crucially, neither RC-S nor RC-T requires a reference video for evaluation, making them more practical for real-world applications where ground truth data may be scarce or non-existent. The PROVE system is available as an open-source PyTorch repository under the Apache 2.0 license, featuring a command-line interface script named run_prove_metrics.py. It requires Python 3.10 or later, PyTorch version 2.6 or higher, Transformers version 4.51 or above, and DINOv2-giant weights. The system mandates the use of masks, where white pixels indicate the object to be removed. The team highlights the deployability of PROVE as an evaluation harness, rather than a direct product feature. For teams with a single GPU, such as those with a mask pipeline, PROVE offers practical performance. On a single RTX 4090 GPU, RC-S can process frames at a speed of 134.6 milliseconds per frame, making it feasible for nightly continuous integration (CI) checks, particularly beneficial for startups. Larger enterprises with extensive editing catalogs stand to gain significantly from PROVE's ability to evaluate without paired ground truth data. The potential industries and applications for PROVE are broad, encompassing smartphone gallery and camera applications, short-video editing platforms, e-commerce catalog cleanup, advertising and stock media production, film and visual effects post-production, real estate imagery enhancement, and privacy redaction in mapping archives. PROVE can be utilized for various purposes, including model comparisons, implementing CI gates based on RC-S and RC-T scores, tuning inference parameters or quantization levels without a reference video, filtering training datasets, and shaping reward models for generative AI. The primary adopters are expected to be computer vision researchers and applied AI/ML engineers working on editing teams, followed by MLOps engineers integrating the scoring into CI pipelines, and product managers conducting vendor comparisons. The system is not designed for real-time on-device processing.