Large-scale Transfer Learning For Low-resource Spoken Language Understanding

Abstract

End-to-end Spoken Language Understanding (SLU) models are made increasingly large and complex to achieve the state-ofthe-art accuracy. However, the increased complexity of a model can also introduce high risk of over-fitting, which is a major challenge in SLU tasks due to the limitation of available data. In this paper, we propose an attention-based SLU model together with three encoder enhancement strategies to overcome data sparsity challenge. The first strategy focuses on the transferlearning approach to improve feature extraction capability of the encoder. It is implemented by pre-training the encoder component with a quantity of Automatic Speech Recognition annotated data relying on the standard Transformer architecture and then fine-tuning the SLU model with a small amount of target labelled data. The second strategy adopts multitask learning strategy, the SLU model integrates the speech recognition model by sharing the same underlying encoder, such that improving robustness and

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyjia2020large

Related papers

• Speech-language Pre-training For End-to-end Spoken Language Understanding (2021)9.41
• Unsupervised Transfer Learning For Spoken Language Understanding In Intelligent Agents (2018)0.00
• Using Speech Synthesis To Train End-to-end Spoken Language Understanding Models (2019)9.23
• Style Attuned Pre-training And Parameter Efficient Fine-tuning For Spoken Language Understanding (2020)6.77
• A Study On The Integration Of Pre-trained SSL, ASR, LM And SLU Models For Spoken Language Understanding (2022)8.09
• Data Augmentation For Spoken Language Understanding Via Pretrained Language Models (2020)0.00
• Integrating Pretrained ASR And LM To Perform Sequence Generation For Spoken Language Understanding (2023)5.24
• Recent Advances In End-to-end Spoken Language Understanding (2019)8.09