Towards Speech-to-text Translation Without Speech Recognition

Abstract

We explore the problem of translating speech to text in low-resource scenarios where neither automatic speech recognition (ASR) nor machine translation (MT) are available, but we have training data in the form of audio paired with text translations. We present the first system for this problem applied to a realistic multi-speaker dataset, the CALLHOME Spanish-English speech translation corpus. Our approach uses unsupervised term discovery (UTD) to cluster repeated patterns in the audio, creating a pseudotext, which we pair with translations to create a parallel text and train a simple bag-of-words MT model. We identify the challenges faced by the system, finding that the difficulty of cross-speaker UTD results in low recall, but that our system is still able to correctly translate some content words in test data.

Stats

• citations23
• S2 citations—
• github stars0
• HF likes0
• heat score10.35
• arxiv keybansal2017towards

Related papers

• Weakly Supervised Spoken Term Discovery Using Cross-lingual Side Information (2016)2.26
• Towards Unsupervised Speech-to-text Translation (2018)0.00
• Strategies For Improving Low Resource Speech To Text Translation Relying On Pre-trained ASR Models (2023)5.24
• End-to-end Text-to-speech For Low-resource Languages By Cross-lingual Transfer Learning (2019)0.00
• An Unsupervised Probability Model For Speech-to-translation Alignment Of Low-resource Languages (2016)6.77
• Improving Cascaded Unsupervised Speech Translation With Denoising Back-translation (2023)0.00
• Leveraging Translations For Speech Transcription In Low-resource Settings (2018)6.77
• Leveraging Weakly Supervised Data To Improve End-to-end Speech-to-text Translation (2018)13.05