CAME: Competitively Learning A Mixture-of-experts Model For First-stage Retrieval

Abstract

The first-stage retrieval aims to retrieve a subset of candidate documents from a huge collection both effectively and efficiently. Since various matching patterns can exist between queries and relevant documents, previous work tries to combine multiple retrieval models to find as many relevant results as possible. The constructed ensembles, whether learned independently or jointly, do not care which component model is more suitable to an instance during training. Thus, they cannot fully exploit the capabilities of different types of retrieval models in identifying diverse relevance patterns. Motivated by this observation, in this paper, we propose a Mixture-of-Experts (MoE) model consisting of representative matching experts and a novel competitive learning mechanism to let the experts develop and enhance their expertise during training. Specifically, our MoE model shares the bottom layers to learn common semantic representations and uses differently structured upper layers to represe

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keycai2023came

Related papers

• Investigating Mixture Of Experts In Dense Retrieval (2024)0.00
• Routerretriever: Routing Over A Mixture Of Expert Embedding Models (2024)0.00
• Mixture Of Experts Approaches In Dense Retrieval Tasks (2025)0.95
• Contrastive Learning And Mixture Of Experts Enables Precise Vector Embeddings (2024)0.00
• Semantic Models For The First-stage Retrieval: A Comprehensive Review (2021)14.27
• L^2R: Lifelong Learning For First-stage Retrieval With Backward-compatible Representations (2023)5.24
• Cot-mote: Exploring Contextual Masked Auto-encoder Pre-training With Mixture-of-textual-experts For Passage Retrieval (2023)0.00
• Learning To Retrieve: How To Train A Dense Retrieval Model Effectively And Efficiently (2020)0.00