From Distillation To Hard Negative Sampling: Making Sparse Neural IR Models More Effective

Abstract

Neural retrievers based on dense representations combined with Approximate Nearest Neighbors search have recently received a lot of attention, owing their success to distillation and/or better sampling of examples for training -- while still relying on the same backbone architecture. In the meantime, sparse representation learning fueled by traditional inverted indexing techniques has seen a growing interest, inheriting from desirable IR priors such as explicit lexical matching. While some architectural variants have been proposed, a lesser effort has been put in the training of such models. In this work, we build on SPLADE -- a sparse expansion-based retriever -- and show to which extent it is able to benefit from the same training improvements as dense models, by studying the effect of distillation, hard-negative mining as well as the Pre-trained Language Model initialization. We furthermore study the link between effectiveness and efficiency, on in-domain and zero-shot settings, lea

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyformal2022from

Related papers

• Syneg: Llm-driven Synthetic Hard-negatives For Dense Retrieval (2024)0.00
• Optimizing Dense Retrieval Model Training With Hard Negatives (2021)16.34
• Efficiently Teaching An Effective Dense Retriever With Balanced Topic Aware Sampling (2021)17.07
• Mistral-splade: Llms For Better Learned Sparse Retrieval (2024)0.00
• Sparse And Dense Retrievers Learn Better Together: Joint Sparse-dense Optimization For Text-image Retrieval (2025)0.00
• CSPLADE: Learned Sparse Retrieval With Causal Language Models (2025)0.00
• Exploring \(\ell_0\) Sparsification For Inference-free Sparse Retrievers (2025)4.52
• Learning To Retrieve: How To Train A Dense Retrieval Model Effectively And Efficiently (2020)0.00