Pretrained Semantic Speech Embeddings For End-to-end Spoken Language Understanding Via Cross-modal Teacher-student Learning

Abstract

Spoken language understanding is typically based on pipeline architectures including speech recognition and natural language understanding steps. These components are optimized independently to allow usage of available data, but the overall system suffers from error propagation. In this paper, we propose a novel training method that enables pretrained contextual embeddings to process acoustic features. In particular, we extend it with an encoder of pretrained speech recognition systems in order to construct end-to-end spoken language understanding systems. Our proposed method is based on the teacher-student framework across speech and text modalities that aligns the acoustic and the semantic latent spaces. Experimental results in three benchmarks show that our system reaches the performance comparable to the pipeline architecture without using any training data and outperforms it after fine-tuning with ten examples per class on two out of three benchmarks.

Stats

• citations20
• S2 citations—
• github stars0
• HF likes0
• heat score9.92
• arxiv keydenisov2020pretrained

Related papers

• Tie Your Embeddings Down: Cross-modal Latent Spaces For End-to-end Spoken Language Understanding (2020)9.03
• Towards End-to-end Spoken Language Understanding (2018)14.73
• ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understanding (2020)9.59
• From Audio To Semantics: Approaches To End-to-end Spoken Language Understanding (2018)13.23
• Speech-language Pre-training For End-to-end Spoken Language Understanding (2021)9.41
• Curriculum-based Transfer Learning For An Effective End-to-end Spoken Language Understanding And Domain Portability (2019)10.74
• Leveraging Multilingual Self-supervised Pretrained Models For Sequence-to-sequence End-to-end Spoken Language Understanding (2023)0.00
• Bootstrapping Meaning Through Listening: Unsupervised Learning Of Spoken Sentence Embeddings (2022)2.26