Look, Imagine And Match: Improving Textual-visual Cross-modal Retrieval With Generative Models

Abstract

Textual-visual cross-modal retrieval has been a hot research topic in both computer vision and natural language processing communities. Learning appropriate representations for multi-modal data is crucial for the cross-modal retrieval performance. Unlike existing image-text retrieval approaches that embed image-text pairs as single feature vectors in a common representational space, we propose to incorporate generative processes into the cross-modal feature embedding, through which we are able to learn not only the global abstract features but also the local grounded features. Extensive experiments show that our framework can well match images and sentences with complex content, and achieve the state-of-the-art cross-modal retrieval results on MSCOCO dataset.

Stats

• citations294
• S2 citations—
• github stars0
• HF likes0
• heat score18.52
• arxiv keygu2017look

Related papers

• CART: A Generative Cross-modal Retrieval Framework With Coarse-to-fine Semantic Modeling (2024)3.58
• Deep Multimodal Image-text Embeddings For Automatic Cross-media Retrieval (2020)0.00
• Semcore: A Semantic-enhanced Generative Cross-modal Retrieval Framework With Mllms (2025)0.00
• Generative Cross-modal Retrieval: Memorizing Images In Multimodal Language Models For Retrieval And Beyond (2024)8.35
• Cross-modal RAG: Sub-dimensional Text-to-image Retrieval-augmented Generation (2025)0.00
• Revisiting Cross Modal Retrieval (2018)0.00
• Stacmr: Scene-text Aware Cross-modal Retrieval (2020)10.48
• Tiger: Unifying Text-to-image Generation And Retrieval With Large Multimodal Models (2024)0.00