Liquid AI Releases Bidirectional Encoders for Long Context on CPU

Liquid AI has released two open-weight bidirectional encoder models, LFM2.5-Encoder-230M and LFM2.5-Encoder-350M, which are designed to efficiently process long contexts of up to 8,192 tokens, particularly on central processing units (CPUs) rather than graphics processing units (GPUs).

These encoders are built upon the LFM2 hybrid backbone and are derived from existing LFM2.5 decoder models. The conversion process involves three key architectural changes. First, the causal attention mask, which restricts attention to preceding tokens, is replaced with a bidirectional mask, allowing each token to attend to all other tokens in the sequence. Second, the short convolutions within the LFM2 architecture are made non-causal by employing symmetric center padding, enabling each token's convolution to incorporate information from neighboring tokens on both sides. Third, the models are trained using a masked language modeling (MLM) objective with a 30% mask rate, a denser rate than BERT's 15%, which Liquid AI suggests is beneficial at this scale.

The training of these encoders occurs in two distinct stages. Stage one focuses on establishing general language competence through an MLM objective with a short context window, trained on a large web corpus with sequences of 1,024 tokens. Stage two expands the context window to the full 8,192 tokens, utilizing the complete data mix. This stage aims to strengthen the models' competence in factual understanding, legal contexts, and multilingual capabilities. Architecturally, the backbone interleaves gated short-convolution blocks with grouped-query attention, a design previously detailed in the LFM2 technical report. Both the LFM2.5-Encoder-230M and LFM2.5-Encoder-350M checkpoints feature a hidden size of 1024 and a vocabulary of 65,536 tokens, supporting 15 different languages. The models are released under the LFM Open License v1.0.

Encoders play a crucial role in natural language processing pipelines, typically functioning beneath components such as classifiers, intent routers, safety filters, and personally identifiable information (PII) detectors. These downstream tasks often operate continuously and are increasingly handling longer input sequences, making efficient processing on CPUs a significant advantage. The development by Liquid AI aims to continue the lineage of encoder models, following advancements like BERT and ModernBERT, by offering a cost-effective solution where the computational cost grows more slowly as input lengths increase. The company has provided detailed breakdowns of the conversion process, performance rankings, per-task scores, and CPU latency figures.