On Joint Training With Interfaces For Spoken Language Understanding

Abstract

Spoken language understanding (SLU) systems extract both text transcripts and semantics associated with intents and slots from input speech utterances. SLU systems usually consist of (1) an automatic speech recognition (ASR) module, (2) an interface module that exposes relevant outputs from ASR, and (3) a natural language understanding (NLU) module. Interfaces in SLU systems carry information on text transcriptions or richer information like neural embeddings from ASR to NLU. In this paper, we study how interfaces affect joint-training for spoken language understanding. Most notably, we obtain the state-of-the-art results on the publicly available 50-hr SLURP dataset. We first leverage large-size pretrained ASR and NLU models that are connected by a text interface, and then jointly train both models via a sequence loss function. For scenarios where pretrained models are not utilized, the best results are obtained through a joint sequence loss training using richer neural interfaces. Fi

Stats

• citations8
• S2 citations—
• github stars0
• HF likes0
• heat score7.16
• arxiv keyraju2021on

Related papers

• Speech To Semantics: Improve ASR And NLU Jointly Via All-neural Interfaces (2020)9.03
• SPLAT: Speech-language Joint Pre-training For Spoken Language Understanding (2020)10.35
• Integrating Pretrained ASR And LM To Perform Sequence Generation For Spoken Language Understanding (2023)5.24
• A Study On The Integration Of Pre-trained SSL, ASR, LM And SLU Models For Spoken Language Understanding (2022)8.09
• Joint Automatic Speech Recognition And Structure Learning For Better Speech Understanding (2025)0.00
• Unislu: Unified Spoken Language Understanding From Heterogeneous Cross-task Datasets (2025)0.00
• Do As I Mean, Not As I Say: Sequence Loss Training For Spoken Language Understanding (2021)6.77
• Pre-training For Spoken Language Understanding With Joint Textual And Phonetic Representation Learning (2021)2.26