Analyzing Speaker Information In Self-supervised Models To Improve Zero-resource Speech Processing

Abstract

Contrastive predictive coding (CPC) aims to learn representations of speech by distinguishing future observations from a set of negative examples. Previous work has shown that linear classifiers trained on CPC features can accurately predict speaker and phone labels. However, it is unclear how the features actually capture speaker and phonetic information, and whether it is possible to normalize out the irrelevant details (depending on the downstream task). In this paper, we first show that the per-utterance mean of CPC features captures speaker information to a large extent. Concretely, we find that comparing means performs well on a speaker verification task. Next, probing experiments show that standardizing the features effectively removes speaker information. Based on this observation, we propose a speaker normalization step to improve acoustic unit discovery using K-means clustering of CPC features. Finally, we show that a language model trained on the resulting units achieves som

Stats

• citations16
• S2 citations—
• github stars0
• HF likes0
• heat score9.23
• arxiv keyvanniekerk2021analyzing

Related papers

• Contrastive Predictive Coding Based Feature For Automatic Speaker Verification (2019)0.00
• Guided Contrastive Self-supervised Pre-training For Automatic Speech Recognition (2022)0.00
• Speech Representation Learning Combining Conformer CPC With Deep Cluster For The Zerospeech Challenge 2021 (2021)7.16
• Self-supervised Predictive Coding Models Encode Speaker And Phonetic Information In Orthogonal Subspaces (2023)7.16
• Contrastive Prediction Strategies For Unsupervised Segmentation And Categorization Of Phonemes And Words (2021)9.23
• Segmental Contrastive Predictive Coding For Unsupervised Word Segmentation (2021)0.00
• Analysing The Masked Predictive Coding Training Criterion For Pre-training A Speech Representation Model (2023)4.52
• Variable-rate Hierarchical CPC Leads To Acoustic Unit Discovery In Speech (2022)0.00