Towards Achieving Human Parity On End-to-end Simultaneous Speech Translation Via LLM Agent

Abstract

In this paper, we present Cross Language Agent -- Simultaneous Interpretation, CLASI, a high-quality and human-like Simultaneous Speech Translation (SiST) System. Inspired by professional human interpreters, we utilize a novel data-driven read-write strategy to balance the translation quality and latency. To address the challenge of translating in-domain terminologies, CLASI employs a multi-modal retrieving module to obtain relevant information to augment the translation. Supported by LLMs, our approach can generate error-tolerated translation by considering the input audio, historical context, and retrieved information. Experimental results show that our system outperforms other systems by significant margins. Aligned with professional human interpreters, we evaluate CLASI with a better human evaluation metric, valid information proportion (VIP), which measures the amount of information that can be successfully conveyed to the listeners. In the real-world scenarios, where the speeches

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keycheng2024towards

Related papers

• Efficient And Adaptive Simultaneous Speech Translation With Fully Unidirectional Architecture (2025)2.26
• Simuls2s-llm: Unlocking Simultaneous Inference Of Speech Llms For Speech-to-speech Translation (2025)3.58
• Hearing To Translate: The Effectiveness Of Speech Modality Integration Into Llms (2026)0.00
• Simulsense: Sense-driven Interpreting For Efficient Simultaneous Speech Translation (2025)0.00
• Visualization: The Missing Factor In Simultaneous Speech Translation (2021)0.00
• Zero-resource Speech Translation And Recognition With Llms (2024)3.58
• SLM-S2ST: A Multimodal Language Model For Direct Speech-to-speech Translation (2025)0.00
• Long-form End-to-end Speech Translation Via Latent Alignment Segmentation (2023)0.00