Mixture Of Experts Approaches In Dense Retrieval Tasks

Abstract

Dense Retrieval Models (DRMs) are a prominent development in Information Retrieval (IR). A key challenge with these neural Transformer-based models is that they often struggle to generalize beyond the specific tasks and domains they were trained on. To address this challenge, prior research in IR incorporated the Mixture-of-Experts (MoE) framework within each Transformer layer of a DRM, which, though effective, substantially increased the number of additional parameters. In this paper, we propose a more efficient design, which introduces a single MoE block (SB-MoE) after the final Transformer layer. To assess the retrieval effectiveness of SB-MoE, we perform an empirical evaluation across three IR tasks. Our experiments involve two evaluation setups, aiming to assess both in-domain effectiveness and the model's zero-shot generalizability. In the first setup, we fine-tune SB-MoE with four different underlying DRMs on seven IR benchmarks and evaluate them on their respective test sets. I

Stats

• citations0
• S2 citations—
• github stars2
• HF likes0
• heat score0.95
• arxiv keysokli2025mixture

Related papers

• Investigating Mixture Of Experts In Dense Retrieval (2024)0.00
• Routerretriever: Routing Over A Mixture Of Expert Embedding Models (2024)0.00
• Interpreting Dense Retrieval As Mixture Of Topics (2021)0.00
• CAME: Competitively Learning A Mixture-of-experts Model For First-stage Retrieval (2023)6.34
• Zero-shot Dense Retrieval With Momentum Adversarial Domain Invariant Representations (2021)6.77
• Mamba Retriever: Utilizing Mamba For Effective And Efficient Dense Retrieval (2024)4.52
• Contrastive Learning And Mixture Of Experts Enables Precise Vector Embeddings (2024)0.00
• Beyond Instruction-conditioning, Mote: Mixture Of Task Experts For Multi-task Embedding Models (2025)0.00