Do Neural Network Cross-modal Mappings Really Bridge Modalities?

Abstract

Feed-forward networks are widely used in cross-modal applications to bridge modalities by mapping distributed vectors of one modality to the other, or to a shared space. The predicted vectors are then used to perform e.g., retrieval or labeling. Thus, the success of the whole system relies on the ability of the mapping to make the neighborhood structure (i.e., the pairwise similarities) of the predicted vectors akin to that of the target vectors. However, whether this is achieved has not been investigated yet. Here, we propose a new similarity measure and two ad hoc experiments to shed light on this issue. In three cross-modal benchmarks we learn a large number of language-to-vision and vision-to-language neural network mappings (up to five layers) using a rich diversity of image and text features and loss functions. Our results reveal that, surprisingly, the neighborhood structure of the predicted vectors consistently resembles more that of the input vectors than that of the target ve

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keycollell2018do

Related papers

• Do Cross Modal Systems Leverage Semantic Relationships? (2019)7.16
• Is Cross-modal Information Retrieval Possible Without Training? (2023)0.00
• Multimodal Representation Alignment For Cross-modal Information Retrieval (2025)0.00
• Cross-modal Scene Networks (2016)14.35
• Preserving Semantic Neighborhoods For Robust Cross-modal Retrieval (2020)10.07
• Adversarial Cross-modal Retrieval Via Learning And Transferring Single-modal Similarities (2019)8.60
• On The Spectral Geometry Of Cross-modal Representations: A Functional Map Diagnostic For Multimodal Alignment (2026)0.00
• Discriminative Semantic Transitive Consistency For Cross-modal Learning (2021)0.00