Leveraging Multilingual Transfer For Unsupervised Semantic Acoustic Word Embeddings

Abstract

Acoustic word embeddings (AWEs) are fixed-dimensional vector representations of speech segments that encode phonetic content so that different realisations of the same word have similar embeddings. In this paper we explore semantic AWE modelling. These AWEs should not only capture phonetics but also the meaning of a word (similar to textual word embeddings). We consider the scenario where we only have untranscribed speech in a target language. We introduce a number of strategies leveraging a pre-trained multilingual AWE model -- a phonetic AWE model trained on labelled data from multiple languages excluding the target. Our best semantic AWE approach involves clustering word segments using the multilingual AWE model, deriving soft pseudo-word labels from the cluster centroids, and then training a Skipgram-like model on the soft vectors. In an intrinsic word similarity task measuring semantics, this multilingual transfer approach outperforms all previous semantic AWE methods. We also sho

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyjacobs2023leveraging

Related papers

• Supervised Acoustic Embeddings And Their Transferability Across Languages (2023)0.00
• Improved Acoustic Word Embeddings For Zero-resource Languages Using Multilingual Transfer (2020)7.81
• Layer-wise Analysis Of Self-supervised Acoustic Word Embeddings: A Study On Speech Emotion Recognition (2024)0.00
• Do Acoustic Word Embeddings Capture Phonological Similarity? An Empirical Study (2021)4.52
• Improving Acoustic Word Embeddings Through Correspondence Training Of Self-supervised Speech Representations (2024)0.00
• Analyzing Acoustic Word Embeddings From Pre-trained Self-supervised Speech Models (2022)9.03
• Asymmetric Proxy Loss For Multi-view Acoustic Word Embeddings (2022)2.26
• Multilingual Acoustic Word Embedding Models For Processing Zero-resource Languages (2020)8.09