A Novel Multimodal Dynamic Fusion Network For Disfluency Detection In Spoken Utterances

Abstract

Disfluency, though originating from human spoken utterances, is primarily studied as a uni-modal text-based Natural Language Processing (NLP) task. Based on early-fusion and self-attention-based multimodal interaction between text and acoustic modalities, in this paper, we propose a novel multimodal architecture for disfluency detection from individual utterances. Our architecture leverages a multimodal dynamic fusion network that adds minimal parameters over an existing text encoder commonly used in prior art to leverage the prosodic and acoustic cues hidden in speech. Through experiments, we show that our proposed model achieves state-of-the-art results on the widely used English Switchboard for disfluency detection and outperforms prior unimodal and multimodal systems in literature by a significant margin. In addition, we make a thorough qualitative analysis and show that, unlike text-only systems, which suffer from spurious correlations in the data, our system overcomes this proble

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyghosh2022a

Related papers

• Streaming Joint Speech Recognition And Disfluency Detection (2022)0.00
• Multi-modal Automated Speech Scoring Using Attention Fusion (2020)0.00
• Span Classification With Structured Information For Disfluency Detection In Spoken Utterances (2022)2.41
• Robust Cross-domain Disfluency Detection With Pattern Match Networks (2018)0.00
• AMFFCN: Attentional Multi-layer Feature Fusion Convolution Network For Audio-visual Speech Enhancement (2021)0.00
• Multi-modal Transformers Utterance-level Code-switching Detection (2020)0.00
• Modality Dropout For Multimodal Device Directed Speech Detection Using Verbal And Non-verbal Features (2023)0.00
• Active Speaker Detection As A Multi-objective Optimization With Uncertainty-based Multimodal Fusion (2021)7.50