Building Robust Spoken Language Understanding By Cross Attention Between Phoneme Sequence And ASR Hypothesis

Abstract

Building Spoken Language Understanding (SLU) robust to Automatic Speech Recognition (ASR) errors is an essential issue for various voice-enabled virtual assistants. Considering that most ASR errors are caused by phonetic confusion between similar-sounding expressions, intuitively, leveraging the phoneme sequence of speech can complement ASR hypothesis and enhance the robustness of SLU. This paper proposes a novel model with Cross Attention for SLU (denoted as CASLU). The cross attention block is devised to catch the fine-grained interactions between phoneme and word embeddings in order to make the joint representations catch the phonetic and semantic features of input simultaneously and for overcoming the ASR errors in downstream natural language understanding (NLU) tasks. Extensive experiments are conducted on three datasets, showing the effectiveness and competitiveness of our approach. Additionally, We also validate the universality of CASLU and prove its complementarity when combin

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keywang2022building

Related papers

• Contrastive Learning For Improving ASR Robustness In Spoken Language Understanding (2022)6.34
• ML-LMCL: Mutual Learning And Large-margin Contrastive Learning For Improving ASR Robustness In Spoken Language Understanding (2023)0.00
• Towards ASR Robust Spoken Language Understanding Through In-context Learning With Word Confusion Networks (2024)0.00
• Integrating Pretrained ASR And LM To Perform Sequence Generation For Spoken Language Understanding (2023)5.24
• Multimodal Audio-textual Architecture For Robust Spoken Language Understanding (2023)0.00
• Unislu: Unified Spoken Language Understanding From Heterogeneous Cross-task Datasets (2025)0.00
• Modality Confidence Aware Training For Robust End-to-end Spoken Language Understanding (2023)2.26
• Do As I Mean, Not As I Say: Sequence Loss Training For Spoken Language Understanding (2021)6.77