Beyond Instruction-conditioning, Mote: Mixture Of Task Experts For Multi-task Embedding Models

Abstract

Dense embeddings are fundamental to modern machine learning systems, powering Retrieval-Augmented Generation (RAG), information retrieval, and representation learning. While instruction-conditioning has become the dominant approach for embedding specialization, its direct application to low-capacity models imposes fundamental representational constraints that limit the performance gains derived from specialization. In this paper, we analyze these limitations and introduce the Mixture of Task Experts (MoTE) transformer block, which leverages task-specialized parameters trained with Task-Aware Contrastive Learning (\tacl) to enhance the model ability to generate specialized embeddings. Empirical results show that MoTE achieves \(64%\) higher performance gains in retrieval datasets (\(+3.27 \rightarrow +5.21\)) and \(43%\) higher performance gains across all datasets (\(+1.81 \rightarrow +2.60\)). Critically, these gains are achieved without altering instructions, training data, inference

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyromero2025beyond

Related papers

• Contrastive Learning And Mixture Of Experts Enables Precise Vector Embeddings (2024)0.00
• Mixture Of Experts Approaches In Dense Retrieval Tasks (2025)0.95
• Cot-mote: Exploring Contextual Masked Auto-encoder Pre-training With Mixture-of-textual-experts For Passage Retrieval (2023)0.00
• Investigating Mixture Of Experts In Dense Retrieval (2024)0.00
• Clip-moe: Towards Building Mixture Of Experts For CLIP With Diversified Multiplet Upcycling (2024)2.26
• MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders Are Better Dense Retrievers (2022)9.97
• M3-embedding: Multi-linguality, Multi-functionality, Multi-granularity Text Embeddings Through Self-knowledge Distillation (2024)19.54
• Improving Embedding With Contrastive Fine-tuning On Small Datasets With Expert-augmented Scores (2024)0.00