Streaming Joint Speech Recognition And Disfluency Detection

Abstract

Disfluency detection has mainly been solved in a pipeline approach, as post-processing of speech recognition. In this study, we propose Transformer-based encoder-decoder models that jointly solve speech recognition and disfluency detection, which work in a streaming manner. Compared to pipeline approaches, the joint models can leverage acoustic information that makes disfluency detection robust to recognition errors and provide non-verbal clues. Moreover, joint modeling results in low-latency and lightweight inference. We investigate two joint model variants for streaming disfluency detection: a transcript-enriched model and a multi-task model. The transcript-enriched model is trained on text with special tags indicating the starting and ending points of the disfluent part. However, it has problems with latency and standard language model adaptation, which arise from the additional disfluency tags. We propose a multi-task model to solve such problems, which has two output layers at the

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfutami2022streaming

Related papers

• Transformer Transducer: A Streamable Speech Recognition Model With Transformer Encoders And RNN-T Loss (2020)18.58
• Transformer Transducer: One Model Unifying Streaming And Non-streaming Speech Recognition (2020)0.00
• Conv-transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-end Speech Recognition (2020)11.08
• Developing Real-time Streaming Transformer Transducer For Speech Recognition On Large-scale Dataset (2020)0.00
• Streaming Decoder-only Automatic Speech Recognition With Discrete Speech Units: A Pilot Study (2024)4.52
• Teaching BERT To Wait: Balancing Accuracy And Latency For Streaming Disfluency Detection (2022)5.24
• Multitask Learning And Joint Optimization For Transformer-rnn-transducer Speech Recognition (2020)8.09
• End-to-end Speech Recognition And Disfluency Removal With Acoustic Language Model Pretraining (2023)0.00