Improving End-to-end Speech Translation By Imitation-based Knowledge Distillation With Synthetic Transcripts

Abstract

End-to-end automatic speech translation (AST) relies on data that combines audio inputs with text translation outputs. Previous work used existing large parallel corpora of transcriptions and translations in a knowledge distillation (KD) setup to distill a neural machine translation (NMT) into an AST student model. While KD allows using larger pretrained models, the reliance of previous KD approaches on manual audio transcripts in the data pipeline restricts the applicability of this framework to AST. We present an imitation learning approach where a teacher NMT system corrects the errors of an AST student without relying on manual transcripts. We show that the NMT teacher can recover from errors in automatic transcriptions and is able to correct erroneous translations of the AST student, leading to improvements of about 4 BLEU points over the standard AST end-to-end baseline on the English-German CoVoST-2 and MuST-C datasets, respectively. Code and data are publicly available.\footnot

Stats

• citations0
• S2 citations—
• github stars1
• HF likes0
• heat score0.60
• arxiv keyhubert2023improving

Related papers

• End-to-end Speech Translation With Knowledge Distillation (2019)0.00
• Harnessing Indirect Training Data For End-to-end Automatic Speech Translation: Tricks Of The Trade (2019)0.00
• Source And Target Bidirectional Knowledge Distillation For End-to-end Speech Translation (2021)9.03
• Decouple Non-parametric Knowledge Distillation For End-to-end Speech Translation (2023)0.00
• Sample, Translate, Recombine: Leveraging Audio Alignments For Data Augmentation In End-to-end Speech Translation (2022)7.81
• Leveraging Weakly Supervised Data To Improve End-to-end Speech-to-text Translation (2018)13.05
• Inter-kd: Intermediate Knowledge Distillation For Ctc-based Automatic Speech Recognition (2022)7.50
• Leveraging Unsupervised And Weakly-supervised Data To Improve Direct Speech-to-speech Translation (2022)8.35