Cross-modal Attribute Insertions For Assessing The Robustness Of Vision-and-language Learning

Abstract

The robustness of multimodal deep learning models to realistic changes in the input text is critical for their applicability to important tasks such as text-to-image retrieval and cross-modal entailment. To measure robustness, several existing approaches edit the text data, but do so without leveraging the cross-modal information present in multimodal data. Information from the visual modality, such as color, size, and shape, provide additional attributes that users can include in their inputs. Thus, we propose cross-modal attribute insertions as a realistic perturbation strategy for vision-and-language data that inserts visual attributes of the objects in the image into the corresponding text (e.g., "girl on a chair" to "little girl on a wooden chair"). Our proposed approach for cross-modal attribute insertions is modular, controllable, and task-agnostic. We find that augmenting input text using cross-modal insertions causes state-of-the-art approaches for text-to-image retrieval and

Stats

• citations0
• S2 citations—
• github stars9
• HF likes0
• heat score2.00
• arxiv keyramshetty2023cross

Related papers

• Object-aware Query Perturbation For Cross-modal Image-text Retrieval (2024)6.52
• Mixgen: A New Multi-modal Data Augmentation (2022)14.47
• Robustness Analysis Of Video-language Models Against Visual And Language Perturbations (2022)5.24
• Self-enhancement Improves Text-image Retrieval In Foundation Visual-language Models (2023)1.56
• Cross-modal Retrieval Augmentation For Multi-modal Classification (2021)9.23
• Webly Supervised Joint Embedding For Cross-modal Image-text Retrieval (2018)13.17
• Preserving Semantic Neighborhoods For Robust Cross-modal Retrieval (2020)10.07
• Cross-modal Image Retrieval With Deep Mutual Information Maximization (2021)9.59