Contrastive Learning And Mixture Of Experts Enables Precise Vector Embeddings

Abstract

The advancement of transformer neural networks has significantly elevated the capabilities of sentence similarity models, but they still struggle with highly discriminative tasks and may produce sub-optimal representations of important documents like scientific literature. With the increased reliance on retrieval augmentation and search, representing diverse documents as concise and descriptive vectors is crucial. This paper improves upon the vectors embeddings of scientific text by assembling niche datasets using co-citations as a similarity metric, focusing on biomedical domains. We apply a novel Mixture of Experts (MoE) extension pipeline to pretrained BERT models, where every multi-layer perceptron section is enlarged and copied into multiple distinct experts. Our MoE variants perform well over \(N\) scientific domains with \(N\) dedicated experts, whereas standard BERT models excel in only one domain at a time. Notably, extending just a single transformer block to MoE captures 85%

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhallee2024contrastive

Related papers

• Investigating Mixture Of Experts In Dense Retrieval (2024)0.00
• Beyond Instruction-conditioning, Mote: Mixture Of Task Experts For Multi-task Embedding Models (2025)0.00
• Mixture Of Experts Approaches In Dense Retrieval Tasks (2025)0.95
• CAME: Competitively Learning A Mixture-of-experts Model For First-stage Retrieval (2023)6.34
• Improving Embedding With Contrastive Fine-tuning On Small Datasets With Expert-augmented Scores (2024)0.00
• Mixture Of Experts With Soft Nearest Neighbor Loss: Resolving Expert Collapse Via Representation Disentanglement (2026)0.00
• Cot-mote: Exploring Contextual Masked Auto-encoder Pre-training With Mixture-of-textual-experts For Passage Retrieval (2023)0.00
• Effective Distributed Representations For Academic Expert Search (2020)6.34