Navigating The Minefield Of MT Beam Search In Cascaded Streaming Speech Translation

Abstract

We adapt the well-known beam-search algorithm for machine translation to operate in a cascaded real-time speech translation system. This proved to be more complex than initially anticipated, due to four key challenges: (1) real-time processing of intermediate and final transcriptions with incomplete words from ASR, (2) emitting intermediate and final translations with minimal user perceived latency, (3) handling beam search hypotheses that have unequal length and different model state, and (4) handling sentence boundaries. Previous work in the field of simultaneous machine translation only implemented greedy decoding. We present a beam-search realization that handles all of the above, providing guidance through the minefield of challenges. Our approach increases the BLEU score by 1 point compared to greedy search, reduces the CPU time by up to 40% and character flicker rate by 20+% compared to a baseline heuristic that just retranslates input repeatedly.

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyrabatin2024navigating

Related papers

• Streaming Parallel Transducer Beam Search With Fast-slow Cascaded Encoders (2022)0.00
• Vectorization Of Hypotheses And Speech For Faster Beam Search In Encoder Decoder-based Speech Recognition (2018)0.00
• Textless Streaming Speech-to-speech Translation Using Semantic Speech Tokens (2024)3.58
• Integration Of Frame- And Label-synchronous Beam Search For Streaming Encoder-decoder Speech Recognition (2023)0.00
• A Fully Differentiable Beam Search Decoder (2019)0.00
• Robust Beam Search For Encoder-decoder Attention Based Speech Recognition Without Length Bias (2020)4.52
• Segment-level Vectorized Beam Search Based On Partially Autoregressive Inference (2023)0.00
• Direct Simultaneous Speech-to-text Translation Assisted By Synchronized Streaming ASR (2021)6.77