Speech-language Pre-training For End-to-end Spoken Language Understanding

Abstract

End-to-end (E2E) spoken language understanding (SLU) can infer semantics directly from speech signal without cascading an automatic speech recognizer (ASR) with a natural language understanding (NLU) module. However, paired utterance recordings and corresponding semantics may not always be available or sufficient to train an E2E SLU model in a real production environment. In this paper, we propose to unify a well-optimized E2E ASR encoder (speech) and a pre-trained language model encoder (language) into a transformer decoder. The unified speech-language pre-trained model (SLP) is continually enhanced on limited labeled data from a target domain by using a conditional masked language model (MLM) objective, and thus can effectively generate a sequence of intent, slot type, and slot value for given input speech in the inference. The experimental results on two public corpora show that our approach to E2E SLU is superior to the conventional cascaded method. It also outperforms the present

Stats

• citations17
• S2 citations—
• github stars0
• HF likes0
• heat score9.41
• arxiv keyqian2021speech

Related papers

• Understanding Semantics From Speech Through Pre-training (2019)0.00
• Integrating Pretrained ASR And LM To Perform Sequence Generation For Spoken Language Understanding (2023)5.24
• End-to-end Spoken Language Understanding Using Transformer Networks And Self-supervised Pre-trained Features (2020)5.24
• End-to-end Spoken Language Understanding For Generalized Voice Assistants (2021)6.34
• Recent Advances In End-to-end Spoken Language Understanding (2019)8.09
• Pre-training For Spoken Language Understanding With Joint Textual And Phonetic Representation Learning (2021)2.26
• Improving End-to-end Models For Set Prediction In Spoken Language Understanding (2022)0.00
• Leveraging Multilingual Self-supervised Pretrained Models For Sequence-to-sequence End-to-end Spoken Language Understanding (2023)0.00