Streaming Multi-speaker ASR With RNN-T

Abstract

Recent research shows end-to-end ASR systems can recognize overlapped speech from multiple speakers. However, all published works have assumed no latency constraints during inference, which does not hold for most voice assistant interactions. This work focuses on multi-speaker speech recognition based on a recurrent neural network transducer (RNN-T) that has been shown to provide high recognition accuracy at a low latency online recognition regime. We investigate two approaches to multi-speaker model training of the RNN-T: deterministic output-target assignment and permutation invariant training. We show that guiding separation with speaker order labels in the former case enhances the high-level speaker tracking capability of RNN-T. Apart from that, with multistyle training on single- and multi-speaker utterances, the resulting models gain robustness against ambiguous numbers of speakers during inference. Our best model achieves a WER of 10.2% on simulated 2-speaker LibriSpeech data, w

Stats

• citations21
• S2 citations—
• github stars0
• HF likes0
• heat score10.07
• arxiv keysklyar2020streaming

Related papers

• Multi-turn RNN-T For Streaming Recognition Of Multi-party Speech (2021)8.82
• Elevating Robust Multi-talker ASR By Decoupling Speaker Separation And Speech Recognition (2025)0.00
• Alignment Restricted Streaming Recurrent Neural Network Transducer (2020)11.19
• Streaming Multi-talker Speech Recognition With Joint Speaker Identification (2021)7.50
• End-to-end Monaural Multi-speaker ASR System Without Pretraining (2018)11.93
• Continuous Streaming Multi-talker ASR With Dual-path Transducers (2021)7.50
• Survey Of End-to-end Multi-speaker Automatic Speech Recognition For Monaural Audio (2025)2.26
• End-to-end Multi-speaker Speech Recognition Using Speaker Embeddings And Transfer Learning (2019)9.41