Lamra: Large Multimodal Model As Your Advanced Retrieval Assistant

Abstract

With the rapid advancement of multimodal information retrieval, increasingly complex retrieval tasks have emerged. Existing methods predominately rely on task-specific fine-tuning of vision-language models, often those trained with image-text contrastive learning. In this paper, we explore the possibility of re-purposing generative Large Multimodal Models (LMMs) for retrieval. This approach enables unifying all retrieval tasks under the same formulation and, more importantly, allows for extrapolation towards unseen retrieval tasks without additional training. Our contributions can be summarised in the following aspects: (i) We introduce LamRA, a versatile framework designed to empower LMMs with sophisticated retrieval and reranking capabilities. (ii) For retrieval, we adopt a two-stage training strategy comprising language-only pre-training and multimodal instruction tuning to progressively enhance LMM's retrieval performance. (iii) For reranking, we employ joint training for both poin

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keyliu2024lamra

Related papers

• Mm-embed: Universal Multimodal Retrieval With Multimodal Llms (2024)0.00
• RETLLM: Training And Data-free Mllms For Multimodal Information Retrieval (2026)1.57
• MLLM Is A Strong Reranker: Advancing Multimodal Retrieval-augmented Generation Via Knowledge-enhanced Reranking And Noise-injected Training (2024)9.18
• Indexing Multimodal Language Models For Large-scale Image Retrieval (2026)0.00
• Retrieval-augmented Multimodal Language Modeling (2022)0.00
• CREM: Compression-driven Representation Enhancement For Multimodal Retrieval And Comprehension (2026)0.00
• LMAR: Language Model Augmented Retriever For Domain-specific Knowledge Indexing (2025)1.57
• Generative Cross-modal Retrieval: Memorizing Images In Multimodal Language Models For Retrieval And Beyond (2024)8.35