An Analysis Of Semantically-aligned Speech-text Embeddings

Abstract

Embeddings play an important role in end-to-end solutions for multi-modal language processing problems. Although there has been some effort to understand the properties of single-modality embedding spaces, particularly that of text, their cross-modal counterparts are less understood. In this work, we study some intrinsic properties of a joint speech-text embedding space, constructed by minimizing the distance between paired utterance and transcription inputs in a teacher-student model setup, that are informative for several prominent use cases. We found that incorporating automatic speech recognition through both pretraining and multitask scenarios aid semantic alignment significantly, resulting in more tightly coupled embeddings. To analyse cross-modal embeddings we utilise a quantitative retrieval accuracy metric for semantic alignment, zero-shot classification for generalisability, and probing of the encoders to observe the extent of knowledge transfer from one modality to another.

Stats

• citations10
• S2 citations—
• github stars0
• HF likes0
• heat score7.81
• arxiv keyhuzaifah2022an

Related papers

• Multimodal Representation Alignment For Cross-modal Information Retrieval (2025)0.00
• Image Search Using Multilingual Texts: A Cross-modal Learning Approach Between Image And Text (2019)0.00
• Joint Wasserstein Autoencoders For Aligning Multimodal Embeddings (2019)7.16
• Structured Multi-modal Feature Embedding And Alignment For Image-sentence Retrieval (2021)12.87
• Learning To Embed Semantic Similarity For Joint Image-text Retrieval (2022)7.50
• Learning Modality-invariant Representations For Speech And Images (2017)8.09
• Webly Supervised Joint Embedding For Cross-modal Image-text Retrieval (2018)13.17
• Modest-align: Data-efficient Alignment For Vision-language Models (2025)0.00