Continuous Streaming Multi-talker ASR With Dual-path Transducers

Abstract

Streaming recognition of multi-talker conversations has so far been evaluated only for 2-speaker single-turn sessions. In this paper, we investigate it for multi-turn meetings containing multiple speakers using the Streaming Unmixing and Recognition Transducer (SURT) model, and show that naively extending the single-turn model to this harder setting incurs a performance penalty. As a solution, we propose the dual-path (DP) modeling strategy first used for time-domain speech separation. We experiment with LSTM and Transformer based DP models, and show that they improve word error rate (WER) performance while yielding faster convergence. We also explore training strategies such as chunk width randomization and curriculum learning for these models, and demonstrate their importance through ablation studies. Finally, we evaluate our models on the LibriCSS meeting data, where they perform competitively with offline separation-based methods.

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keyraj2021continuous

Related papers

• Streaming End-to-end Multi-talker Speech Recognition (2020)11.49
• Streaming Multi-talker Speech Recognition With Joint Speaker Identification (2021)7.50
• Multi-turn RNN-T For Streaming Recognition Of Multi-party Speech (2021)8.82
• Separator-transducer-segmenter: Streaming Recognition And Segmentation Of Multi-party Speech (2022)0.00
• Unified Modeling Of Multi-talker Overlapped Speech Recognition And Diarization With A Sidecar Separator (2023)7.50
• Streaming Multi-speaker ASR With RNN-T (2020)10.07
• A Comparative Study On Speaker-attributed Automatic Speech Recognition In Multi-party Meetings (2022)8.09
• A Comparative Study On Multichannel Speaker-attributed Automatic Speech Recognition In Multi-party Meetings (2022)5.24