Orthros: Non-autoregressive End-to-end Speech Translation With Dual-decoder

Abstract

Fast inference speed is an important goal towards real-world deployment of speech translation (ST) systems. End-to-end (E2E) models based on the encoder-decoder architecture are more suitable for this goal than traditional cascaded systems, but their effectiveness regarding decoding speed has not been explored so far. Inspired by recent progress in non-autoregressive (NAR) methods in text-based translation, which generates target tokens in parallel by eliminating conditional dependencies, we study the problem of NAR decoding for E2E-ST. We propose a novel NAR E2E-ST framework, Orthros, in which both NAR and autoregressive (AR) decoders are jointly trained on the shared speech encoder. The latter is used for selecting better translation among various length candidates generated from the former, which dramatically improves the effectiveness of a large length beam with negligible overhead. We further investigate effective length prediction methods from speech inputs and the impact of voca

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keyinaguma2020orthros

Related papers

• A Comparative Study On Non-autoregressive Modelings For Speech-to-text Generation (2021)11.76
• Fast-md: Fast Multi-decoder End-to-end Speech Translation With Non-autoregressive Hidden Intermediates (2021)7.16
• Improving Non-autoregressive End-to-end Speech Recognition With Pre-trained Acoustic And Language Models (2022)10.07
• Synchronous Speech Recognition And Speech-to-text Translation With Interactive Decoding (2019)10.48
• TSNAT: Two-step Non-autoregressvie Transformer Models For Speech Recognition (2021)10.61
• A Non-autoregressive Generation Framework For End-to-end Simultaneous Speech-to-speech Translation (2024)6.34
• Non-autoregressive End-to-end Approaches For Joint Automatic Speech Recognition And Spoken Language Understanding (2023)5.84
• Improving Cross-lingual Transfer Learning For End-to-end Speech Recognition With Speech Translation (2020)9.92