GME: Improving Universal Multimodal Retrieval By Multimodal Llms

Abstract

Universal Multimodal Retrieval (UMR) aims to enable search across various modalities using a unified model, where queries and candidates can consist of pure text, images, or a combination of both. Previous work has attempted to adopt multimodal large language models (MLLMs) to realize UMR using only text data. However, our preliminary experiments demonstrate that more diverse multimodal training data can further unlock the potential of MLLMs. Despite its effectiveness, the existing multimodal training data is highly imbalanced in terms of modality, which motivates us to develop a training data synthesis pipeline and construct a large-scale, high-quality fused-modal training dataset. Based on the synthetic training data, we develop the General Multimodal Embedder (GME), an MLLM-based dense retriever designed for UMR. Furthermore, we construct a comprehensive UMR Benchmark (UMRB) to evaluate the effectiveness of our approach. Experimental results show that our method achieves state-of-th

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2024gme

Related papers

• Mm-embed: Universal Multimodal Retrieval With Multimodal Llms (2024)0.00
• U-MARVEL: Unveiling Key Factors For Universal Multimodal Retrieval Via Embedding Learning With Mllms (2025)3.11
• RETLLM: Training And Data-free Mllms For Multimodal Information Retrieval (2026)1.57
• CREM: Compression-driven Representation Enhancement For Multimodal Retrieval And Comprehension (2026)0.00
• Modality Curation: Building Universal Embeddings For Advanced Multimodal Information Retrieval (2025)0.00
• Beyond Global Similarity: Towards Fine-grained, Multi-condition Multimodal Retrieval (2026)2.20
• Lamra: Large Multimodal Model As Your Advanced Retrieval Assistant (2024)7.50
• Generalized Contrastive Learning For Universal Multimodal Retrieval (2025)0.00