Cascaded Models With Cyclic Feedback For Direct Speech Translation

Abstract

Direct speech translation describes a scenario where only speech inputs and corresponding translations are available. Such data are notoriously limited. We present a technique that allows cascades of automatic speech recognition (ASR) and machine translation (MT) to exploit in-domain direct speech translation data in addition to out-of-domain MT and ASR data. After pre-training MT and ASR, we use a feedback cycle where the downstream performance of the MT system is used as a signal to improve the ASR system by self-training, and the MT component is fine-tuned on multiple ASR outputs, making it more tolerant towards spelling variations. A comparison to end-to-end speech translation using components of identical architecture and the same data shows gains of up to 3.8 BLEU points on LibriVoxDeEn and up to 5.1 BLEU points on CoVoST for German-to-English speech translation.

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keylam2020cascaded

Related papers

• Tight Integrated End-to-end Training For Cascaded Speech Translation (2020)8.35
• When End-to-end Is Overkill: Rethinking Cascaded Speech-to-text Translation (2025)0.00
• Improving Cascaded Unsupervised Speech Translation With Denoising Back-translation (2023)0.00
• Harnessing Indirect Training Data For End-to-end Automatic Speech Translation: Tricks Of The Trade (2019)0.00
• Transformer-based Cascaded Multimodal Speech Translation (2019)0.00
• Direct Simultaneous Speech-to-text Translation Assisted By Synchronized Streaming ASR (2021)6.77
• A Comparative Study On End-to-end Speech To Text Translation (2019)11.67
• Instance-based Model Adaptation For Direct Speech Translation (2019)0.00