Meta Releases Muse Voice Transcribe for Real-Time Audio Processing

Meta Superintelligence Labs announced the release of Muse Voice Transcribe this week, a novel system designed to consolidate three distinct voice processing tasks into a single, unified model. Traditionally, production voice stacks rely on separate systems for automatic speech recognition (ASR), speaker diarization, and endpoint detection, with each component introducing latency and potential failure points during data hand-offs. Muse Voice Transcribe, described by Meta as its first real-time audio perception model, integrates streaming ASR, speaker diarization capable of distinguishing over 20 speakers, and endpointing into one continuous process. This single-pass approach eliminates the need for post-processing, aiming to reduce latency and improve reliability.

The Muse Voice Transcribe model is built upon the Muse Spark family and operates as an autoregressive multimodal system. Audio input is processed in 80-millisecond chunks at a rate of 12.5 Hz, with each chunk converted into a single soft token. Following the processing of each audio chunk, the model makes a binary decision: it can predict a <|next_audio|> token, signaling it should continue listening, or it can emit a text token representing transcribed speech. When the <|next_audio|> token is predicted, it is subsequently replaced by the next incoming audio chunk. Upon the conclusion of the audio stream, an <|empty_audio|> token is introduced, prompting the model to output any remaining transcribed text without requiring further audio input. This architecture allows for synchronized listening and transcription within a single decoder loop, thereby avoiding potential drift issues associated with separate alignment stages.

A key innovation in Muse Voice Transcribe is its adaptive delay mechanism, trained using reinforcement learning. The model's ability to control when it listens also dictates the amount of audio context preceding each transcribed word, a gap Meta refers to as 'delay.' While a longer delay generally leads to more accurate transcripts and higher latency, Meta has trained the model to dynamically manage this trade-off. The reinforcement learning process combines rewards for minimizing word error rate with rewards for minimizing delay. This results in a policy that adjusts the delay on a per-word basis, adapting to the perceived difficulty of the speech. Meta reports that this adaptive approach enhances performance by optimizing the balance between accuracy and responsiveness.

Muse Voice Transcribe is currently accessible as a hosted API through the Meta Model API. The service is priced at $3.00 per 1,000 audio minutes, which equates to approximately $0.18 per hour. This new model already underpins dictation functionalities within Meta AI for Mac and the Muse Code product. Meta has not released the model weights, meaning a self-hosted deployment option is not currently available to users. The development represents a significant step in Meta's efforts to advance real-time audio processing capabilities, integrating complex functionalities into a more efficient and streamlined system.