Contrastive Separative Coding For Self-supervised Representation Learning

Abstract

To extract robust deep representations from long sequential modeling of speech data, we propose a self-supervised learning approach, namely Contrastive Separative Coding (CSC). Our key finding is to learn such representations by separating the target signal from contrastive interfering signals. First, a multi-task separative encoder is built to extract shared separable and discriminative embedding; secondly, we propose a powerful cross-attention mechanism performed over speaker representations across various interfering conditions, allowing the model to focus on and globally aggregate the most critical information to answer the "query" (current bottom-up embedding) while paying less attention to interfering, noisy, or irrelevant parts; lastly, we form a new probabilistic contrastive loss which estimates and maximizes the mutual information between the representations and the global speaker vector. While most prior unsupervised methods have focused on predicting the future, neighboring,

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2021contrastive

Related papers

• Speaker Representation Learning Via Contrastive Loss With Maximal Speaker Separability (2022)10.68
• Unsupervised Speech Segmentation And Variable Rate Representation Learning Using Segmental Contrastive Predictive Coding (2021)9.92
• Learning Problem-agnostic Speech Representations From Multiple Self-supervised Tasks (2019)15.54
• Aligned Contrastive Predictive Coding (2021)9.23
• Segmental Contrastive Predictive Coding For Unsupervised Word Segmentation (2021)0.00
• Self-supervised Learning From Contrastive Mixtures For Personalized Speech Enhancement (2020)0.00
• Self-supervised Predictive Coding Models Encode Speaker And Phonetic Information In Orthogonal Subspaces (2023)7.16
• Ccc-wav2vec 2.0: Clustering Aided Cross Contrastive Self-supervised Learning Of Speech Representations (2022)7.81