DeepSeek AI Releases 1M Context LLM With FP4 KV Cache

DeepSeek AI released DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts large language model designed to address the input-heavy workload and significant KV cache strain associated with long-horizon agents and million-token contexts. The model boasts a 552 billion backbone parameter count and an additional 196 billion Engram parameters, enabling it to handle a 1 million token context window. During operation, DeepSeek-V4.1-Flash activates 8 billion parameters per token during the prefill phase and 16 billion parameters per token during decoding. A key innovation is its global KV cache footprint, which is reported to be 890 bytes per token. This represents a substantial reduction, being approximately one-quarter of the footprint of DeepSeek-V4-Flash and roughly 437 times smaller than that of DeepSeek-V1.

The model's architecture incorporates a 40-layer backbone divided into a 20-layer causal encoder and a 20-layer decoder. Inspired by the YOCO architecture, the decoder does not compute its own global KV. Instead, per-layer projection weights derive the global KV from the final encoder hidden state. This design choice means that prompt tokens terminate at the encoder, which effectively halves the prefill compute required. The model also implements sliding-window attention (SWA) with a 128-token window in every layer. Decoder SWA states are reconstructed by replaying only the last 128 prompt tokens, a technique the research team refers to as Decoder SWA Bounded Replay.

DeepSeek-V4.1-Flash utilizes a refined version of Compressed Sparse Attention (CSA2), building upon the CSA and Heavily Compressed Attention techniques used in DeepSeek-V4. This CSA2 implementation targets cache size reduction along the layer axis. Each CSA2 layer is assigned one of three static modes: 'Full', 'Reindex', or 'Reuse'. In 'Full' mode, a layer computes its own main KV and projects the indexer K from it, selecting the top 512 indices. The 'Reindex' mode reuses the main KV and indexer K from the last 'Full' layer but rescores them with its own indexer Q. The 'Reuse' mode goes further by reusing both the main KV and the latest Top-K indices, completely skipping the indexer computation. Each layer independently maintains its own main Q and SWA KV. The model features 18 CSA2 encoder layers.

DeepSeek-V4.1-Flash is made accessible through open weights released under an MIT license. It is available via vLLM, SGLang, and Transformers integration paths on Hugging Face. Additionally, DeepSeek AI has announced a public API offering distinct reasoning tiers: low, high, and max. This release aims to provide a more efficient and deployable solution for handling extensive context windows in LLM applications.