Towards Unsupervised Speaker Diarization System For Multilingual Telephone Calls Using Pre-trained Whisper Model And Mixture Of Sparse Autoencoders

Abstract

Existing speaker diarization systems typically rely on large amounts of manually annotated data, which is labor-intensive and difficult to obtain, especially in real-world scenarios. Additionally, language-specific constraints in these systems significantly hinder their effectiveness and scalability in multilingual settings. In this paper, we propose a cluster-based speaker diarization system designed for multilingual telephone call applications. Our proposed system supports multiple languages and eliminates the need for large-scale annotated data during training by utilizing the multilingual Whisper model to extract speaker embeddings. Additionally, we introduce a network architecture called Mixture of Sparse Autoencoders (Mix-SAE) for unsupervised speaker clustering. Experimental results on the evaluation dataset derived from two-speaker subsets of benchmark CALLHOME and CALLFRIEND telephonic speech corpora demonstrate the superior performance of the proposed Mix-SAE network to other

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keylam2024towards

Related papers

• Unified Modeling Of Multi-talker Overlapped Speech Recognition And Diarization With A Sidecar Separator (2023)7.50
• Adapting Diarization-conditioned Whisper For End-to-end Multi-talker Speech Recognition (2025)0.00
• Semi-supervised Multi-channel Speaker Diarization With Cross-channel Attention (2023)2.26
• Data Efficient Child-adult Speaker Diarization With Simulated Conversations (2024)0.00
• A Study Of Semi-supervised Speaker Diarization System Using Gan Mixture Model (2019)0.00
• Dicow: Diarization-conditioned Whisper For Target Speaker Automatic Speech Recognition (2024)8.09
• Speaker Conditioned Acoustic Modeling For Multi-speaker Conversational ASR (2021)4.52
• Multi-scale Speaker Diarization With Neural Affinity Score Fusion (2020)6.77