Thinking Machines Lab Releases Inkling: 975B Parameter Open-Weights Multimodal MoE

Thinking Machines Lab released Inkling, a 975 billion parameter open-weights multimodal Mixture-of-Experts (MoE) model, on an unspecified recent date. This model features 41 billion active parameters and supports a context window of up to 1 million tokens. Inkling was trained on 45 trillion tokens encompassing text, images, audio, and video, accepting text, image, and audio inputs, with UTF-8 text as its sole output format. The lab also previewed Inkling-Small, a 276 billion parameter MoE with 12 billion active parameters, which reportedly matches or surpasses its larger counterpart on several benchmarks.

The architecture of Inkling is a 66-layer decoder-only transformer utilizing a sparse MoE feed-forward backbone. Each MoE layer contains 256 routed experts and 2 shared experts, with six routed experts and both shared experts activating per token. A sigmoid-based router manages expert selection, employing an auxiliary-loss-free load-balancing bias. The model's attention mechanism interleaves sliding-window and global layers at a 5:1 ratio, incorporating 8 KV heads. It uses relative positional embeddings instead of RoPE and applies short convolutions after key and value projections and on residual branch outputs.

Inkling's multimodality is achieved without an encoder. Audio inputs are processed as dMel spectrograms, and images are converted into 40x40 pixel patches via a four-layer hMLP. These modalities are then projected by a lightweight embedding layer and processed jointly with text tokens by the decoder. Training of Inkling utilized Muon for large matrix weights and Adam for other parameters, conducted on NVIDIA GB300 NVL72 systems. Post-training involved bootstrapping from supervised fine-tuning (SFT) on synthetic data.