Towards Supervised Performance On Speaker Verification With Self-supervised Learning By Leveraging Large-scale ASR Models

Abstract

Recent advancements in Self-Supervised Learning (SSL) have shown promising results in Speaker Verification (SV). However, narrowing the performance gap with supervised systems remains an ongoing challenge. Several studies have observed that speech representations from large-scale ASR models contain valuable speaker information. This work explores the limitations of fine-tuning these models for SV using an SSL contrastive objective in an end-to-end approach. Then, we propose a framework to learn speaker representations in an SSL context by fine-tuning a pre-trained WavLM with a supervised loss using pseudo-labels. Initial pseudo-labels are derived from an SSL DINO-based model and are iteratively refined by clustering the model embeddings. Our method achieves 0.99% EER on VoxCeleb1-O, establishing the new state-of-the-art on self-supervised SV. As this performance is close to our supervised baseline of 0.94% EER, this contribution is a step towards supervised performance on SV with SSL.

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keymiara2024towards

Related papers

• One-step Knowledge Distillation And Fine-tuning In Using Large Pre-trained Self-supervised Learning Models For Speaker Verification (2023)7.81
• Why Does Self-supervised Learning For Speech Recognition Benefit Speaker Recognition? (2022)10.74
• Large-scale Self-supervised Speech Representation Learning For Automatic Speaker Verification (2021)15.25
• Efficient Infusion Of Self-supervised Representations In Automatic Speech Recognition (2024)0.00
• Additive Margin In Contrastive Self-supervised Frameworks To Learn Discriminative Speaker Representations (2024)2.26
• C3-DINO: Joint Contrastive And Non-contrastive Self-supervised Learning For Speaker Verification (2022)10.21
• Asymmetric Clean Segments-guided Self-supervised Learning For Robust Speaker Verification (2023)5.84
• Fine-tuning Strategies For Faster Inference Using Speech Self-supervised Models: A Comparative Study (2023)8.35