Multimodal Clustering With Role Induced Constraints For Speaker Diarization

Abstract

Speaker clustering is an essential step in conventional speaker diarization systems and is typically addressed as an audio-only speech processing task. The language used by the participants in a conversation, however, carries additional information that can help improve the clustering performance. This is especially true in conversational interactions, such as business meetings, interviews, and lectures, where specific roles assumed by interlocutors (manager, client, teacher, etc.) are often associated with distinguishable linguistic patterns. In this paper we propose to employ a supervised text-based model to extract speaker roles and then use this information to guide an audio-based spectral clustering step by imposing must-link and cannot-link constraints between segments. The proposed method is applied on two different domains, namely on medical interactions and on podcast episodes, and is shown to yield improved results when compared to the audio-only approach.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyflemotomos2022multimodal

Related papers

• Integrating Audio, Visual, And Semantic Information For Enhanced Multimodal Speaker Diarization (2024)0.00
• Robust Speaker Clustering Using Mixtures Of Von Mises-fisher Distributions For Naturalistic Audio Streams (2018)4.52
• Enhancements For Audio-only Diarization Systems (2019)0.00
• Highly Efficient Real-time Streaming And Fully On-device Speaker Diarization With Multi-stage Clustering (2022)0.00
• Assessing The Robustness Of Spectral Clustering For Deep Speaker Diarization (2024)3.58
• Improving Speaker Diarization Using Semantic Information: Joint Pairwise Constraints Propagation (2023)0.00
• End-to-end Speaker Diarization As Post-processing (2020)11.08
• Multi-class Spectral Clustering With Overlaps For Speaker Diarization (2020)10.35