Towards The Evaluation Of Automatic Simultaneous Speech Translation From A Communicative Perspective

Abstract

In recent years, automatic speech-to-speech and speech-to-text translation has gained momentum thanks to advances in artificial intelligence, especially in the domains of speech recognition and machine translation. The quality of such applications is commonly tested with automatic metrics, such as BLEU, primarily with the goal of assessing improvements of releases or in the context of evaluation campaigns. However, little is known about how the output of such systems is perceived by end users or how they compare to human performances in similar communicative tasks. In this paper, we present the results of an experiment aimed at evaluating the quality of a real-time speech translation engine by comparing it to the performance of professional simultaneous interpreters. To do so, we adopt a framework developed for the assessment of human interpreters and use it to perform a manual evaluation on both human and machine performances. In our sample, we found better performance for the human

Stats

• citations17
• S2 citations—
• github stars0
• HF likes0
• heat score9.41
• arxiv keyfantinuoli2021towards

Related papers

• End-to-end Evaluation For Low-latency Simultaneous Speech Translation (2023)0.00
• Evaluating The IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, And Segmentation (2024)0.00
• Visualization: The Missing Factor In Simultaneous Speech Translation (2021)0.00
• How To Evaluate Speech Translation With Source-aware Neural MT Metrics (2025)0.00
• Automatic Quality Assessment For Speech Translation Using Joint ASR And MT Features (2016)4.52
• Towards Achieving Human Parity On End-to-end Simultaneous Speech Translation Via LLM Agent (2024)0.00
• MT Metrics Correlate With Human Ratings Of Simultaneous Speech Translation (2022)3.58
• Better Late Than Never: Meta-evaluation Of Latency Metrics For Simultaneous Speech-to-text Translation (2025)1.81