Simuls2s-llm: Unlocking Simultaneous Inference Of Speech Llms For Speech-to-speech Translation
2025 · Keqi Deng, Wenxi Chen, Xie Chen, et al.
Abstract
Simultaneous speech translation (SST) outputs translations in parallel with streaming speech input, balancing translation quality and latency. While large language models (LLMs) have been extended to handle the speech modality, streaming remains challenging as speech is prepended as a prompt for the entire generation process. To unlock LLM streaming capability, this paper proposes SimulS2S-LLM, which trains speech LLMs offline and employs a test-time policy to guide simultaneous inference. SimulS2S-LLM alleviates the mismatch between training and inference by extracting boundary-aware speech prompts that allows it to be better matched with text input data. SimulS2S-LLM achieves simultaneous speech-to-speech translation (Simul-S2ST) by predicting discrete output speech tokens and then synthesising output speech using a pre-trained vocoder. An incremental beam search is designed to expand the search space of speech token prediction without increasing latency. Experiments on the CVSS spee
Authors
(none)
Tags
Stats
Related papers
- Streamspeech: Simultaneous Speech-to-speech Translation With Multi-task Learning (2024)7.81
- Efficient And Adaptive Simultaneous Speech Translation With Fully Unidirectional Architecture (2025)2.26
- SLM-S2ST: A Multimodal Language Model For Direct Speech-to-speech Translation (2025)0.00
- Label-synchronous Neural Transducer For E2E Simultaneous Speech Translation (2024)0.00
- Learning When To Speak: Latency And Quality Trade-offs For Simultaneous Speech-to-speech Translation With Offline Models (2023)0.00
- Simulsense: Sense-driven Interpreting For Efficient Simultaneous Speech Translation (2025)0.00
- Towards Achieving Human Parity On End-to-end Simultaneous Speech Translation Via LLM Agent (2024)0.00
- MSLM-S2ST: A Multitask Speech Language Model For Textless Speech-to-speech Translation With Speaker Style Preservation (2024)0.00