Phonetic-and-semantic Embedding Of Spoken Words With Applications In Spoken Content Retrieval

Abstract

Word embedding or Word2Vec has been successful in offering semantics for text words learned from the context of words. Audio Word2Vec was shown to offer phonetic structures for spoken words (signal segments for words) learned from signals within spoken words. This paper proposes a two-stage framework to perform phonetic-and-semantic embedding on spoken words considering the context of the spoken words. Stage 1 performs phonetic embedding with speaker characteristics disentangled. Stage 2 then performs semantic embedding in addition. We further propose to evaluate the phonetic-and-semantic nature of the audio embeddings obtained in Stage 2 by parallelizing with text embeddings. In general, phonetic structure and semantics inevitably disturb each other. For example the words "brother" and "sister" are close in semantics but very different in phonetic structure, while the words "brother" and "bother" are in the other way around. But phonetic-and-semantic embedding is attractive, as shown

Stats

• citations19
• S2 citations—
• github stars0
• HF likes0
• heat score9.76
• arxiv keychen2018phonetic

Related papers

• Spoken Word2vec: Learning Skipgram Embeddings From Speech (2023)3.58
• Improved Audio Embeddings By Adjacency-based Clustering With Applications In Spoken Term Detection (2018)0.00
• Speech2vec: A Sequence-to-sequence Framework For Learning Word Embeddings From Speech (2018)14.15
• Learning Word Embeddings From Speech (2017)0.00
• Segmental Audio Word2vec: Representing Utterances As Sequences Of Vectors With Applications In Spoken Term Detection (2018)11.08
• Audio Word2vec: Unsupervised Learning Of Audio Segment Representations Using Sequence-to-sequence Autoencoder (2016)0.00
• Acoustic Word Embedding System For Code-switching Query-by-example Spoken Term Detection (2020)3.58
• Language Transfer Of Audio Word2vec: Learning Audio Segment Representations Without Target Language Data (2017)3.58