Joint Speaker Counting, Speech Recognition, And Speaker Identification For Overlapped Speech Of Any Number Of Speakers

Abstract

We propose an end-to-end speaker-attributed automatic speech recognition model that unifies speaker counting, speech recognition, and speaker identification on monaural overlapped speech. Our model is built on serialized output training (SOT) with attention-based encoder-decoder, a recently proposed method for recognizing overlapped speech comprising an arbitrary number of speakers. We extend SOT by introducing a speaker inventory as an auxiliary input to produce speaker labels as well as multi-speaker transcriptions. All model parameters are optimized by speaker-attributed maximum mutual information criterion, which represents a joint probability for overlapped speech recognition and speaker identification. Experiments on LibriSpeech corpus show that our proposed method achieves significantly better speaker-attributed word error rate than the baseline that separately performs overlapped speech recognition and speaker identification.

Stats

• citations46
• S2 citations—
• github stars0
• HF likes0
• heat score12.54
• arxiv keykanda2020joint

Related papers

• Investigation Of End-to-end Speaker-attributed ASR For Continuous Multi-talker Recordings (2020)10.35
• End-to-end Multi-speaker Speech Recognition Using Speaker Embeddings And Transfer Learning (2019)9.41
• Unified Autoregressive Modeling For Joint End-to-end Multi-talker Overlapped Speech Recognition And Speaker Attribute Estimation (2021)6.34
• Unified Modeling Of Multi-talker Overlapped Speech Recognition And Diarization With A Sidecar Separator (2023)7.50
• Streaming Multi-talker Speech Recognition With Joint Speaker Identification (2021)7.50
• A Toolkit For Joint Speaker Diarization And Identification With Application To Speaker-attributed ASR (2024)0.00
• A Comparative Study Of Modular And Joint Approaches For Speaker-attributed ASR On Monaural Long-form Audio (2021)7.50
• A Comparative Study On Speaker-attributed Automatic Speech Recognition In Multi-party Meetings (2022)8.09