Enhancing End-to-end Conversational Speech Translation Through Target Language Context Utilization

Abstract

Incorporating longer context has been shown to benefit machine translation, but the inclusion of context in end-to-end speech translation (E2E-ST) remains under-studied. To bridge this gap, we introduce target language context in E2E-ST, enhancing coherence and overcoming memory constraints of extended audio segments. Additionally, we propose context dropout to ensure robustness to the absence of context, and further improve performance by adding speaker information. Our proposed contextual E2E-ST outperforms the isolated utterance-based E2E-ST approach. Lastly, we demonstrate that in conversational speech, contextual information primarily contributes to capturing context style, as well as resolving anaphora and named entities.

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyhussein2023enhancing

Related papers

• Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers (2021)10.07
• Attentive Contextual Carryover For Multi-turn End-to-end Spoken Language Understanding (2021)7.16
• Leveraging Weakly Supervised Data To Improve End-to-end Speech-to-text Translation (2018)13.05
• Cif-based Collaborative Decoding For End-to-end Contextual Speech Recognition (2020)9.76
• Multilingual End-to-end Speech Translation (2019)0.00
• Investigating Context Features Hidden In End-to-end TTS (2018)0.00
• Towards Effective And Compact Contextual Representation For Conformer Transducer Speech Recognition Systems (2023)7.16
• Improving Cross-lingual Transfer Learning For End-to-end Speech Recognition With Speech Translation (2020)9.92