From Audio To Semantics: Approaches To End-to-end Spoken Language Understanding

Abstract

Conventional spoken language understanding systems consist of two main components: an automatic speech recognition module that converts audio to a transcript, and a natural language understanding module that transforms the resulting text (or top N hypotheses) into a set of domains, intents, and arguments. These modules are typically optimized independently. In this paper, we formulate audio to semantic understanding as a sequence-to-sequence problem [1]. We propose and compare various encoder-decoder based approaches that optimize both modules jointly, in an end-to-end manner. Evaluations on a real-world task show that 1) having an intermediate text representation is crucial for the quality of the predicted semantics, especially the intent arguments and 2) jointly optimizing the full system improves overall accuracy of prediction. Compared to independently trained models, our best jointly trained model achieves similar domain and intent prediction F1 scores, but improves argument word

Stats

• citations57
• S2 citations—
• github stars0
• HF likes0
• heat score13.23
• arxiv keyhaghani2018from

Related papers

• Towards End-to-end Spoken Language Understanding (2018)14.73
• Speech To Semantics: Improve ASR And NLU Jointly Via All-neural Interfaces (2020)9.03
• Exploiting Sentence And Context Representations In Deep Neural Models For Spoken Language Understanding (2016)0.00
• Recent Advances In End-to-end Spoken Language Understanding (2019)8.09
• Speech-language Pre-training For End-to-end Spoken Language Understanding (2021)9.41
• Pretrained Semantic Speech Embeddings For End-to-end Spoken Language Understanding Via Cross-modal Teacher-student Learning (2020)9.92
• Modality Confidence Aware Training For Robust End-to-end Spoken Language Understanding (2023)2.26
• Understanding Semantics From Speech Through Pre-training (2019)0.00