Learning Relation Alignment For Calibrated Cross-modal Retrieval

Abstract

Despite the achievements of large-scale multimodal pre-training approaches, cross-modal retrieval, e.g., image-text retrieval, remains a challenging task. To bridge the semantic gap between the two modalities, previous studies mainly focus on word-region alignment at the object level, lacking the matching between the linguistic relation among the words and the visual relation among the regions. The neglect of such relation consistency impairs the contextualized representation of image-text pairs and hinders the model performance and the interpretability. In this paper, we first propose a novel metric, Intra-modal Self-attention Distance (ISD), to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. In response, we present Inter-modal Alignment on Intra-modal Self-attentions (IAIS), a regularized training method to optimize the ISD and calibrate intra-modal self-attentions from the two modalities mutually via inter-modal alignment

Stats

• citations14
• S2 citations—
• github stars0
• HF likes0
• heat score8.82
• arxiv keyren2021learning

Related papers

• Cross-modal Implicit Relation Reasoning And Aligning For Text-to-image Person Retrieval (2023)18.15
• Towards Fast And Accurate Image-text Retrieval With Self-supervised Fine-grained Alignment (2023)11.99
• Multimodal Representation Alignment For Cross-modal Information Retrieval (2025)0.00
• Correspondence-free Domain Alignment For Unsupervised Cross-domain Image Retrieval (2023)9.23
• Cross-modal And Uni-modal Soft-label Alignment For Image-text Retrieval (2024)15.75
• Modest-align: Data-efficient Alignment For Vision-language Models (2025)0.00
• Multimodal Representation Learning Conditioned On Semantic Relations (2025)0.00
• Contextblip: Doubly Contextual Alignment For Contrastive Image Retrieval From Linguistically Complex Descriptions (2024)0.00