Linkup Research Releases SPARSEUP Sparse Embedding Model

Linkup Research has released SPARSEUP, an open-source learned sparse embedding model designed to enhance information retrieval systems. The model is built upon a 149 million-parameter ModernBERT backbone and is distributed under the permissive Apache 2.0 license, making it freely available for use and modification. According to Linkup Research, SPARSEUP achieved an average nDCG@10 score of 56.4 on the BEIR-13 benchmark, a widely recognized collection of retrieval tasks. The research team claims this performance represents the strongest publicly available sparse encoder under 150 million parameters that utilizes a public vocabulary. The model is readily deployable, with its weights accessible on Hugging Face. It can be loaded and utilized through popular libraries like Transformers or Sentence Transformers by setting the `trust_remote_code=True` parameter.

The development of SPARSEUP was prompted by the recent release of DenseOn and LateOn models by LightOn. LightOn provided open data, a training methodology, a dense model, and a late-interaction model. SPARSEUP aims to fill the gap in the retrieval landscape by offering a robust sparse model that can be directly compared with these other approaches, given its use of the same backbone family and fine-tuning data. This allows for a comprehensive side-by-side comparison of dense, late-interaction, and sparse retrieval techniques.

The construction of SPARSEUP involved fine-tuning a model that started from the LateOn-unsupervised checkpoint. This checkpoint lacked a Masked Language Model (MLM) head, so the Linkup team re-integrated the original MLM head from ModernBERT. The fine-tuning process employed LightOn's fine-tuning mixture, focusing exclusively on contrastive learning. During training, each query was paired with 7 hard negatives sampled from a pool of 50, in addition to in-batch negatives. Notably, the training process did not involve cross-encoder distillation and was completed on a single NVIDIA H100 GPU. A standard SPLADE model on the same backbone produced an excessive number of stopword tokens, a problem that Linkup addressed through three key modifications.

To mitigate the issue of excessive stopwords and improve the model's efficiency, Linkup implemented three specific changes. First, logit shifting was applied, where the encoder computes `log(1 + ReLU(x - 15))`. This adjustment prevents ModernBERT's MLM logits from becoming too high, which could saturate the logarithm and lead to dense token bags at the start of training. Second, a per-position top-k mechanism was introduced, ensuring that each input token retains only its 12 strongest vocabulary dimensions before undergoing max pooling. This caps the expansion of dimensions per token rather than limiting the overall vector size. Finally, case folding was implemented. Byte-level BPE tokenization can treat variations like 'heat', 'Heat', 'Ġheat', and 'ĠHeat' as distinct IDs. SPARSEUP unifies these variations into a single ID, further refining the model's output and reducing redundancy.