Automatic Quality Assessment For Speech Translation Using Joint ASR And MT Features

Abstract

This paper addresses automatic quality assessment of spoken language translation (SLT). This relatively new task is defined and formalized as a sequence labeling problem where each word in the SLT hypothesis is tagged as good or bad according to a large feature set. We propose several word confidence estimators (WCE) based on our automatic evaluation of transcription (ASR) quality, translation (MT) quality, or both (combined ASR+MT). This research work is possible because we built a specific corpus which contains 6.7k utterances for which a quintuplet containing: ASR output, verbatim transcript, text translation, speech translation and post-edition of translation is built. The conclusion of our multiple experiments using joint ASR and MT features for WCE is that MT features remain the most influent while ASR feature can bring interesting complementary information. Our robust quality estimators for SLT can be used for re-scoring speech translation graphs or for providing feedback to the

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keyle2016automatic

Related papers

• Towards The Evaluation Of Automatic Simultaneous Speech Translation From A Communicative Perspective (2021)9.41
• How To Evaluate Speech Translation With Source-aware Neural MT Metrics (2025)0.00
• Evaluating The IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, And Segmentation (2024)0.00
• A Reference-less Quality Metric For Automatic Speech Recognition Via Contrastive-learning Of A Multi-language Model With Self-supervision (2023)2.51
• An Evaluation Of Word-level Confidence Estimation For End-to-end Automatic Speech Recognition (2021)0.00
• Ccatmos: Convolutional Context-aware Transformer Network For Non-intrusive Speech Quality Assessment (2022)5.24
• Leveraging Weakly Supervised Data To Improve End-to-end Speech-to-text Translation (2018)13.05
• Simultaneous Translation For Unsegmented Input: A Sliding Window Approach (2022)0.00