Moshi
Moshi is a speech-text foundation model and full-duplex spoken dialogue framework from Kyutai, released on September 18, 2024. Kyutai describes it as the first real-time full-duplex spoken large language model, with a theoretical latency of 160 ms (200 ms in practice).
Released
September 18, 2024
Type
audio
License
open-weight
CC-BY 4.0
Context
—
Capabilities
Architecture
Parameters: 7B (Temporal Transformer)
Sources
Every figure on this page was checked against these pages on . A figure we could not source is left out.
- Kyutai blog index (opens in a new tab)
- Hugging Face model card (kyutai org): Moshi (opens in a new tab)
- GitHub: kyutai-labs/moshi (opens in a new tab)
- Kyutai: Moshi (opens in a new tab)
Spotted something out of date? Tell us.
Links
In the news: Moshi and Kyutai
MarkTechPost
Kyutai Releases Voice of Reason: A Speech-Native Model that Solves Spoken Math with Reinforcement Learning
Kyutai has released Voice of Reason, a pair of open-weight speech-to-speech models designed to solve mathematical problems spoken aloud. These models operate without an intermediate transcription step or a separate text-based large language model (LLM) in the processing loop, directly converting spoken input into spoken mathematical reasoning and solutions. Both models are built upon the GLM-4-Voice-9B architecture and incorporate supervised fine-tuning (SFT) and reinforcement learning (RL) techniques. The direct speech-to-speech approach aims to overcome limitations of cascaded pipelines, which involve speech-to-text, text LLM, and text-to-speech stages. While cascaded systems currently lead in reasoning accuracy, each stage introduces latency and can result in the loss of crucial paralinguistic cues such as tone of voice. Speech-native models, to maintain interactivity, must generate audio output at regular intervals, which constrains the number of internal reasoning tokens they can process. The base GLM-4-Voice model achieved an accuracy of 27.3% on the GSM8K spoken math benchmark. A previous method, STITCH, improved this to 58.7% by incorporating reasoning chunks. Kyutai's research team claims their work represents the first application of reinforcement learning to mathematical reasoning within speech-native AI models. The training process for Voice of Reason involves two stages. The first stage, supervised fine-tuning (SFT), utilized 150,616 problems sourced from Orca-Math. Each problem was rewritten for speech by Qwen3-235B and then vocalized in various voices using Kyutai's DSM TTS system. This SFT stage alone boosted the GLM-4-Voice model's accuracy from 27.3% to 61.7%. The second stage employs reinforcement learning (RL). For each spoken question, the model generates four potential responses at a temperature of 0.9. A separate judge model, Qwen3-235B-A22B-2507, evaluates the decoded text of these responses with a binary reward, without access to the correct answer. This judge model demonstrated an 88% agreement rate with human evaluations on 100 hand-checked cases. The rewards are centered within each group, forming a group-relative REINFORCE objective, which is similar to GRPO but omits PPO clipping and KL regularization. The training was conducted on 16 H100 GPUs, involving approximately 1,500 RL updates. Voice of Reason achieves a significant accuracy of 77.1% on spoken GSM8K, a substantial improvement over the base model's 27.3%. The models are available for self-hosting, with both BF16 checkpoints having been run on a single H100 GPU. Users also require the GLM-4-Voice repository for its speech tokenizer and decoder. The weights are distributed under the GLM-4-Voice license, and as of the release, no inference providers on Hugging Face host these models. The GLM-4-Voice model interleaves its output, producing 13 text tokens followed by 26 audio tokens in a repeating pattern.
MarkTechPost
Gradium Launches Voice Design: Write a Prompt, Get a Brand New Synthetic Voice in Seconds
Paris-based Gradium, a voice AI company that originated from the Kyutai research lab, has introduced Voice Design, a novel system capable of generating entirely new synthetic voices based on written descriptions. This technology addresses a common bottleneck for voice agent teams who often require specific vocal characteristics not present in existing voice catalogs. Traditionally, fulfilling such requests involved either finding a suitable voice from a limited selection or employing voice cloning, a process that necessitates sourcing audio, obtaining consent, and securing licensing for each individual speaker. Voice Design bypasses these complexities by accepting a textual description as its sole input, producing complete, ready-to-use voices in a matter of seconds without requiring any reference audio or speaker identification. The Voice Design system is now accessible through the Gradium API and its Studio platform, integrated into all subscription tiers, including the free plan. The generated voices utilize the same streaming Text-to-Speech (TTS) endpoint as Gradium's catalog voices, maintaining consistent latency and output formats. The core of the system lies in its ability to interpret detailed casting briefs provided as text. Gradium's documentation outlines a comprehensive list of attributes the model responds to, encompassing elements such as gender, age range, accent or origin, pitch, speaking pace, energy level, vocal timbre and resonance, speech register, delivery mannerisms, and the specific role the voice is intended to perform. These descriptions can range from 1 to 500 characters and are supported in English, French, Spanish, Portuguese, and German. Gradium recommends concluding the prompt with the intended use case, as this information significantly influences the voice's delivery and register, going beyond mere tonal characteristics. Upon submitting a prompt, Voice Design typically returns one to five candidate voices, which are variations of a single character, usually within 3 to 5 seconds. To generate a different character, a new, distinct description must be provided, rather than requesting additional samples of the same character. The process of moving from a candidate to a production-ready voice involves a sequence of four API calls. Initially, a POST request to `/voice-generator/generate` creates candidate IDs with a 'ready: false' status. Subsequently, a GET request to `/voice-generator/embeddings` is used to poll the system until the 'ready' status for these candidates is updated to 'true'. Each candidate voice can then be auditioned using the standard TTS endpoint by referencing its unique candidate ID as the `voice_id`. Finally, a POST request to `/voices/from-embedding` is used to promote the selected candidate voice for production use. A key distinction highlighted by Gradium is that these generated voices do not carry the same restrictions as cloned voices; for instance, audition text is not capped.
MarkTechPost
Kyutai Releases MuScriptor: An Open-Weight Decoder-Only Transformer for Multi-Instrument Music Transcription to MIDI
Kyutai and Mirelo released MuScriptor this week, an open-weight decoder-only Transformer model designed to transcribe multi-instrument music into MIDI format. Automatic Music Transcription (AMT) has historically struggled with complex mixes, but MuScriptor aims to bridge this gap by training on real, multi-instrument recordings across various genres. The model processes mel-spectrograms of audio segments and autoregressively predicts MIDI tokens for pitch, timing, and instrument, treating transcription as a language modeling task. MuScriptor is available in three weight variants: small (103 million parameters), medium (307 million parameters), and large (1.4 billion parameters). The inference code is released under the MIT license, while the model weights are under the CC BY-NC 4.0 license, restricting commercial use. The development focused on data curation and a three-stage training pipeline. The initial stage involves pre-training on approximately 1.45 million MIDI files from the D<sub>Synth</sub> dataset, synthesized on-the-fly with extensive augmentations like pitch shifting, tempo changes, and instrument randomization using over 250 soundfonts. The second stage, fine-tuning, utilizes D<sub>Real</sub>, an internal dataset comprising 170,000 recordings totaling over 11,000 hours. These recordings feature aligned note annotations, with most alignments generated through audio-symbolic synchronization techniques such as interpolation and dynamic time warping. Pairs with poor alignment, identified by warping distance and maximum time-dilation factors, are filtered out. The final stage employs reinforcement learning post-training using D<sub>RL</sub>, a set of 300 manually verified tracks, applying a GRPO-like method that combines REINFORCE with policy gradient optimization.