RETLLM: Training And Data-free Mllms For Multimodal Information Retrieval

Abstract

Multimodal information retrieval (MMIR) has gained attention for its flexibility in handling text, images, or mixed queries and candidates. Recent breakthroughs in multimodal large language models (MLLMs) boost MMIR performance by incorporating MLLM knowledge under the contrastive finetuning framework. However, they suffer from pre-training inconsistency and require large datasets. In this work, we introduce a novel framework, RetLLM, designed to query MLLMs for MMIR in a training- and data-free manner. Specifically, we formulate MMIR as a similarity score generation task and prompt MLLMs to directly predict retrieval scores in a coarse-then-fine pipeline. At the coarse stage, a top-k filtering strategy builds a small yet high-quality candidate pool for each query, enabling MLLMs to focus on semantically relevant candidates. Subsequently, the retrieval score is predicted by feeding both the query and candidate into MLLMs at the fine stage. Importantly, we propose a visual enhancement m

Stats

• citations0
• S2 citations—
• github stars3
• HF likes0
• heat score1.57
• arxiv keysu2026retllm

Related papers

• Indexing Multimodal Language Models For Large-scale Image Retrieval (2026)0.00
• Mm-embed: Universal Multimodal Retrieval With Multimodal Llms (2024)0.00
• MLLM Is A Strong Reranker: Advancing Multimodal Retrieval-augmented Generation Via Knowledge-enhanced Reranking And Noise-injected Training (2024)9.18
• Lamra: Large Multimodal Model As Your Advanced Retrieval Assistant (2024)7.50
• Freeret: Mllms As Training-free Retrievers (2025)0.00
• Docmmir: A Framework For Document Multi-modal Information Retrieval (2025)3.46
• SLQ: Bridging Modalities Via Shared Latent Queries For Retrieval With Frozen Mllms (2026)0.00
• CREM: Compression-driven Representation Enhancement For Multimodal Retrieval And Comprehension (2026)0.00