CA*: Addressing Evaluation Pitfalls In Computation-aware Latency For Simultaneous Speech Translation

Abstract

Simultaneous speech translation (SimulST) systems must balance translation quality with response time, making latency measurement crucial for evaluating their real-world performance. However, there has been a longstanding belief that current metrics yield unrealistically high latency measurements in unsegmented streaming settings. In this paper, we investigate this phenomenon, revealing its root cause in a fundamental misconception underlying existing latency evaluation approaches. We demonstrate that this issue affects not only streaming but also segment-level latency evaluation across different metrics. Furthermore, we propose a modification to correctly measure computation-aware latency for SimulST systems, addressing the limitations present in existing metrics.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyxu2024ca

Related papers

• Better Late Than Never: Meta-evaluation Of Latency Metrics For Simultaneous Speech-to-text Translation (2025)1.81
• Over-generation Cannot Be Rewarded: Length-adaptive Average Lagging For Simultaneous Speech Translation (2022)7.16
• Visualization: The Missing Factor In Simultaneous Speech Translation (2021)0.00
• Learning When To Speak: Latency And Quality Trade-offs For Simultaneous Speech-to-speech Translation With Offline Models (2023)0.00
• From Start To Finish: Latency Reduction Strategies For Incremental Speech Synthesis In Simultaneous Speech-to-speech Translation (2021)2.26
• End-to-end Evaluation For Low-latency Simultaneous Speech Translation (2023)0.00
• Efficient And Adaptive Simultaneous Speech Translation With Fully Unidirectional Architecture (2025)2.26
• Streamatt: Direct Streaming Speech-to-text Translation With Attention-based Audio History Selection (2024)4.52