Style Attuned Pre-training And Parameter Efficient Fine-tuning For Spoken Language Understanding

Abstract

Neural models have yielded state-of-the-art results in deciphering spoken language understanding (SLU) problems; however, these models require a significant amount of domain-specific labeled examples for training, which is prohibitively expensive. While pre-trained language models like BERT have been shown to capture a massive amount of knowledge by learning from unlabeled corpora and solve SLU using fewer labeled examples for adaption, the encoding of knowledge is implicit and agnostic to downstream tasks. Such encoding results in model inefficiencies in parameter usage: an entirely new model is required for every domain. To address these challenges, we introduce a novel SLU framework, comprising a conversational language modeling (CLM) pre-training task and a light encoder architecture. The CLM pre-training enables networks to capture the representation of the language in conversation style with the presence of ASR errors. The light encoder architecture separates the shared pre-train

Stats

• citations7
• S2 citations—
• github stars0
• HF likes0
• heat score6.77
• arxiv keycao2020style

Related papers

• Understanding Semantics From Speech Through Pre-training (2019)0.00
• Speech-language Pre-training For End-to-end Spoken Language Understanding (2021)9.41
• Multi-domain Spoken Language Understanding Using Domain- And Task-aware Parameterization (2020)3.58
• A Study On The Integration Of Pre-trained SSL, ASR, LM And SLU Models For Spoken Language Understanding (2022)8.09
• Large-scale Transfer Learning For Low-resource Spoken Language Understanding (2020)2.26
• Unsupervised Transfer Learning For Spoken Language Understanding In Intelligent Agents (2018)0.00
• ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understanding (2020)9.59
• Pre-training For Spoken Language Understanding With Joint Textual And Phonetic Representation Learning (2021)2.26