Multimodal Learned Sparse Retrieval For Image Suggestion

Abstract

Learned Sparse Retrieval (LSR) is a group of neural methods designed to encode queries and documents into sparse lexical vectors. These vectors can be efficiently indexed and retrieved using an inverted index. While LSR has shown promise in text retrieval, its potential in multi-modal retrieval remains largely unexplored. Motivated by this, in this work, we explore the application of LSR in the multi-modal domain, i.e., we focus on Multi-Modal Learned Sparse Retrieval (MLSR). We conduct experiments using several MLSR model configurations and evaluate the performance on the image suggestion task. We find that solving the task solely based on the image content is challenging. Enriching the image content with its caption improves the model performance significantly, implying the importance of image captions to provide fine-grained concepts and context information of images. Our approach presents a practical and effective solution for training LSR retrieval models in multi-modal settings.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keynguyen2024multimodal

Related papers

• Sparse And Dense Retrievers Learn Better Together: Joint Sparse-dense Optimization For Text-image Retrieval (2025)0.00
• Indexing Multimodal Language Models For Large-scale Image Retrieval (2026)0.00
• Mm-embed: Universal Multimodal Retrieval With Multimodal Llms (2024)0.00
• CSPLADE: Learned Sparse Retrieval With Causal Language Models (2025)0.00
• Lamra: Large Multimodal Model As Your Advanced Retrieval Assistant (2024)7.50
• Mistral-splade: Llms For Better Learned Sparse Retrieval (2024)0.00
• Combating Visual Neglect And Semantic Drift In Large Multimodal Models For Enhanced Cross-modal Retrieval (2026)0.00
• SLQ: Bridging Modalities Via Shared Latent Queries For Retrieval With Frozen Mllms (2026)0.00