Mamba Retriever: Utilizing Mamba For Effective And Efficient Dense Retrieval

Abstract

In the information retrieval (IR) area, dense retrieval (DR) models use deep learning techniques to encode queries and passages into embedding space to compute their semantic relations. It is important for DR models to balance both efficiency and effectiveness. Pre-trained language models (PLMs), especially Transformer-based PLMs, have been proven to be effective encoders of DR models. However, the self-attention component in Transformer-based PLM results in a computational complexity that grows quadratically with sequence length, and thus exhibits a slow inference speed for long-text retrieval. Some recently proposed non-Transformer PLMs, especially the Mamba architecture PLMs, have demonstrated not only comparable effectiveness to Transformer-based PLMs on generative language tasks but also better efficiency due to linear time scaling in sequence length. This paper implements the Mamba Retriever to explore whether Mamba can serve as an effective and efficient encoder of DR model for

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keyzhang2024mamba

Related papers

• Mixture Of Experts Approaches In Dense Retrieval Tasks (2025)0.95
• Efficiently Teaching An Effective Dense Retriever With Balanced Topic Aware Sampling (2021)17.07
• Learning To Retrieve: How To Train A Dense Retrieval Model Effectively And Efficiently (2020)0.00
• Interpreting Dense Retrieval As Mixture Of Topics (2021)0.00
• Disentangled Modeling Of Domain And Relevance For Adaptable Dense Retrieval (2022)0.00
• Dense Passage Retrieval: Is It Retrieving? (2024)6.34
• Making Large Language Models Efficient Dense Retrievers (2025)0.00
• MUSE: Mamba Is Efficient Multi-scale Learner For Text-video Retrieval (2024)6.34