Hypothesis Clustering And Merging: Novel Multitalker Speech Recognition With Speaker Tokens

Abstract

In many real-world scenarios, such as meetings, multiple speakers are present with an unknown number of participants, and their utterances often overlap. We address these multi-speaker challenges by a novel attention-based encoder-decoder method augmented with special speaker class tokens obtained by speaker clustering. During inference, we select multiple recognition hypotheses conditioned on predicted speaker cluster tokens, and these hypotheses are merged by agglomerative hierarchical clustering (AHC) based on the normalized edit distance. The clustered hypotheses result in the multi-speaker transcriptions with the appropriate number of speakers determined by AHC. Our experiments on the LibriMix dataset demonstrate that our proposed method was particularly effective in complex 3-mix environments, achieving a 55% relative error reduction on clean data and a 36% relative error reduction on noisy data compared with conventional serialized output training.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykashiwagi2024hypothesis

Related papers

• Robust Speaker Clustering Using Mixtures Of Von Mises-fisher Distributions For Naturalistic Audio Streams (2018)4.52
• Unified Modeling Of Multi-talker Overlapped Speech Recognition And Diarization With A Sidecar Separator (2023)7.50
• Streaming Multi-talker Speech Recognition With Joint Speaker Identification (2021)7.50
• A Purely End-to-end System For Multi-speaker Speech Recognition (2018)12.25
• Listening To Multi-talker Conversations: Modular And End-to-end Perspectives (2024)0.00
• Joint Speaker Counting, Speech Recognition, And Speaker Identification For Overlapped Speech Of Any Number Of Speakers (2020)12.54
• Elevating Robust Multi-talker ASR By Decoupling Speaker Separation And Speech Recognition (2025)0.00
• A Sidecar Separator Can Convert A Single-talker Speech Recognition System To A Multi-talker One (2023)9.03