Joint Wasserstein Autoencoders For Aligning Multimodal Embeddings

Abstract

One of the key challenges in learning joint embeddings of multiple modalities, e.g. of images and text, is to ensure coherent cross-modal semantics that generalize across datasets. We propose to address this through joint Gaussian regularization of the latent representations. Building on Wasserstein autoencoders (WAEs) to encode the input in each domain, we enforce the latent embeddings to be similar to a Gaussian prior that is shared across the two domains, ensuring compatible continuity of the encoded semantic representations of images and texts. Semantic alignment is achieved through supervision from matching image-text pairs. To show the benefits of our semi-supervised representation, we apply it to cross-modal retrieval and phrase localization. We not only achieve state-of-the-art accuracy, but significantly better generalization across datasets, owing to the semantic continuity of the latent space.

Stats

• citations8
• S2 citations—
• github stars0
• HF likes0
• heat score7.16
• arxiv keymahajan2019joint

Related papers

• Webly Supervised Joint Embedding For Cross-modal Image-text Retrieval (2018)13.17
• Multimodal Representation Alignment For Cross-modal Information Retrieval (2025)0.00
• Aligning Multilingual Word Embeddings For Cross-modal Retrieval Task (2019)2.26
• Learning To Embed Semantic Similarity For Joint Image-text Retrieval (2022)7.50
• Learning Robust Visual-semantic Embeddings (2017)15.22
• Preserving Semantic Neighborhoods For Robust Cross-modal Retrieval (2020)10.07
• An Analysis Of Semantically-aligned Speech-text Embeddings (2022)7.81
• WAVE: Learning Unified & Versatile Audio-visual Embeddings With Multimodal LLM (2025)0.00