Unsupervised Transfer Learning For Spoken Language Understanding In Intelligent Agents

Abstract

User interaction with voice-powered agents generates large amounts of unlabeled utterances. In this paper, we explore techniques to efficiently transfer the knowledge from these unlabeled utterances to improve model performance on Spoken Language Understanding (SLU) tasks. We use Embeddings from Language Model (ELMo) to take advantage of unlabeled data by learning contextualized word representations. Additionally, we propose ELMo-Light (ELMoL), a faster and simpler unsupervised pre-training method for SLU. Our findings suggest unsupervised pre-training on a large corpora of unlabeled utterances leads to significantly better SLU performance compared to training from scratch and it can even outperform conventional supervised transfer. Additionally, we show that the gains from unsupervised transfer techniques can be further improved by supervised transfer. The improvements are more pronounced in low resource settings and when using only 1000 labeled in-domain samples, our techniques match

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysiddhant2018unsupervised

Related papers

• Large-scale Transfer Learning For Low-resource Spoken Language Understanding (2020)2.26
• Style Attuned Pre-training And Parameter Efficient Fine-tuning For Spoken Language Understanding (2020)6.77
• Speech-language Pre-training For End-to-end Spoken Language Understanding (2021)9.41
• A Study On The Integration Of Pre-trained SSL, ASR, LM And SLU Models For Spoken Language Understanding (2022)8.09
• Data Augmentation For Spoken Language Understanding Via Pretrained Language Models (2020)0.00
• Towards Reducing The Need For Speech Training Data To Build Spoken Language Understanding Systems (2022)8.35
• Understanding Semantics From Speech Through Pre-training (2019)0.00
• Exploring Fine-tuning Of Large Audio Language Models For Spoken Language Understanding Under Limited Speech Data (2025)0.00