H Company Releases NeoMME Multimodal Encoders

H Company has released NeoMME, a new family of multimodal encoders designed to improve the efficiency of visual document retrieval systems. The NeoMME family includes two models: a 260 million parameter version and an 800 million parameter version. These encoders are distinct from previous approaches, which often repurposed generative vision-language models. Such repurposed models typically retain a separately pretrained vision tower and a causal decoder, components that are unnecessary for retrieval tasks and contribute to parameter and compute overhead. NeoMME eliminates these redundant parts, featuring a single Transformer that processes both multilingual text tokens and raw 32x32 RGB image patches through the same layers. Both models were trained from random initialization.

The retrieval-focused fine-tune of NeoMME, named NeoMME-Retriever, has demonstrated strong performance, achieving a 0.523 nDCG@10 score on the ViDoRe v3 benchmark with its 260 million parameter configuration. This efficiency makes the models deployable in production environments. H Company has released all checkpoints under the Apache 2.0 license, with immediate support available in the Hugging Face Transformers library. The 260 million parameter model is capable of indexing 51.3 pages per second on a single NVIDIA L40S GPU and can encode a query in 78.3 milliseconds on a CPU-only host, highlighting its speed and accessibility.

Each NeoMME model utilizes a single tower architecture. Text input is processed through an ALBERT-style factorized embedding, projecting a 256-dimensional lookup to the model's width. Images are divided into non-overlapping 32x32 patches and processed by a 2-layer Multi-Layer Perceptron (MLP) trained from scratch, eschewing patch-merging modules or separate vision towers like SigLIP2. Both models support a substantial context length of 16,384 tokens, which is sufficient to accommodate two standard 4K UHD images (3840x2160 resolution) after they have been patched. The architecture incorporates symmetric sliding-window attention in most layers, with global attention applied in every sixth layer and the final layer. The stack also features grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs. The precise parameter counts for the models are 262,937,906 and 793,715,032.

For text processing, NeoMME employs a whitespace-unconstrained Byte Pair Encoding (BPE) tokenizer with a vocabulary size of 131,072 entries, which was also trained from scratch. This tokenizer is designed to be more efficient, emitting 44.4% fewer tokens compared to ModernBERT across 14 target languages in the FLORES-200 devtest set. The pretraining methodology for NeoMME involves discrete masked diffusion over text, with optional conditioning. This approach allows the model to learn robust representations for both text and image data within a unified framework, contributing to its strong performance in multimodal retrieval tasks.