Object-aware Query Perturbation For Cross-modal Image-text Retrieval

Abstract

The pre-trained vision and language (V\&L) models have substantially improved the performance of cross-modal image-text retrieval. In general, however, V\&L models have limited retrieval performance for small objects because of the rough alignment between words and the small objects in the image. In contrast, it is known that human cognition is object-centric, and we pay more attention to important objects, even if they are small. To bridge this gap between the human cognition and the V\&L model's capability, we propose a cross-modal image-text retrieval framework based on ``object-aware query perturbation.'' The proposed method generates a key feature subspace of the detected objects and perturbs the corresponding queries using this subspace to improve the object awareness in the image. In our proposed method, object-aware cross-modal image-text retrieval is possible while keeping the rich expressive power and retrieval performance of existing V\&L models without additional fine-tunin

Stats

• citations3
• S2 citations—
• github stars9
• HF likes0
• heat score6.52
• arxiv keysogi2024object

Related papers

• Benchmark Granularity And Model Robustness For Image-text Retrieval (2024)0.00
• Cross-modal Attribute Insertions For Assessing The Robustness Of Vision-and-language Learning (2023)2.00
• Object-centric Open-vocabulary Image-retrieval With Aggregated Features (2023)0.00
• Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models And Vision Language Models (2024)8.82
• Cross-modal RAG: Sub-dimensional Text-to-image Retrieval-augmented Generation (2025)0.00
• Vldeformer: Vision-language Decomposed Transformer For Fast Cross-modal Retrieval (2021)10.21
• A Little More Like This: Text-to-image Retrieval With Vision-language Models Using Relevance Feedback (2025)0.00
• Cross-modal Retrieval Augmentation For Multi-modal Classification (2021)9.23