Uniir: Training And Benchmarking Universal Multimodal Information Retrievers

Abstract

Existing information retrieval (IR) models often assume a homogeneous format, limiting their applicability to diverse user needs, such as searching for images with text descriptions, searching for a news article with a headline image, or finding a similar photo with a query image. To approach such different information-seeking demands, we introduce UniIR, a unified instruction-guided multimodal retriever capable of handling eight distinct retrieval tasks across modalities. UniIR, a single retrieval system jointly trained on ten diverse multimodal-IR datasets, interprets user instructions to execute various retrieval tasks, demonstrating robust performance across existing datasets and zero-shot generalization to new tasks. Our experiments highlight that multi-task training and instruction tuning are keys to UniIR's generalization ability. Additionally, we construct the M-BEIR, a multimodal retrieval benchmark with comprehensive results, to standardize the evaluation of universal multimo

Stats

• citations24
• S2 citations—
• github stars0
• HF likes0
• heat score10.48
• arxiv keywei2023uniir

Related papers

• Unihgkr: Unified Instruction-aware Heterogeneous Knowledge Retrievers (2024)0.00
• BEIR: A Heterogenous Benchmark For Zero-shot Evaluation Of Information Retrieval Models (2021)6.67
• Universal Vision-language Dense Retrieval: Learning A Unified Representation Space For Multi-modal Retrieval (2022)3.45
• Modality Curation: Building Universal Embeddings For Advanced Multimodal Information Retrieval (2025)0.00
• MAIR: A Massive Benchmark For Evaluating Instructed Retrieval (2024)6.41
• GME: Improving Universal Multimodal Retrieval By Multimodal Llms (2024)0.00
• Mm-embed: Universal Multimodal Retrieval With Multimodal Llms (2024)0.00
• Unifier: A Unified Retriever For Large-scale Retrieval (2022)7.50