Kyutai Releases MuScriptor for Multi-Instrument Music Transcription

Kyutai and Mirelo released MuScriptor this week, an open-weight decoder-only Transformer model designed to transcribe multi-instrument music into MIDI format. Automatic Music Transcription (AMT) has historically struggled with complex mixes, but MuScriptor aims to bridge this gap by training on real, multi-instrument recordings across various genres. The model processes mel-spectrograms of audio segments and autoregressively predicts MIDI tokens for pitch, timing, and instrument, treating transcription as a language modeling task.

MuScriptor is available in three weight variants: small (103 million parameters), medium (307 million parameters), and large (1.4 billion parameters). The inference code is released under the MIT license, while the model weights are under the CC BY-NC 4.0 license, restricting commercial use. The development focused on data curation and a three-stage training pipeline. The initial stage involves pre-training on approximately 1.45 million MIDI files from the D_Synth dataset, synthesized on-the-fly with extensive augmentations like pitch shifting, tempo changes, and instrument randomization using over 250 soundfonts.

The second stage, fine-tuning, utilizes D_Real, an internal dataset comprising 170,000 recordings totaling over 11,000 hours. These recordings feature aligned note annotations, with most alignments generated through audio-symbolic synchronization techniques such as interpolation and dynamic time warping. Pairs with poor alignment, identified by warping distance and maximum time-dilation factors, are filtered out. The final stage employs reinforcement learning post-training using D_RL, a set of 300 manually verified tracks, applying a GRPO-like method that combines REINFORCE with policy gradient optimization.