Suber: A Metric For Automatic Evaluation Of Subtitle Quality

Abstract

This paper addresses the problem of evaluating the quality of automatically generated subtitles, which includes not only the quality of the machine-transcribed or translated speech, but also the quality of line segmentation and subtitle timing. We propose SubER - a single novel metric based on edit distance with shifts that takes all of these subtitle properties into account. We compare it to existing metrics for evaluating transcription, translation, and subtitle quality. A careful human evaluation in a post-editing scenario shows that the new metric has a high correlation with the post-editing effort and direct human assessment scores, outperforming baseline metrics considering only the subtitle text, such as WER and BLEU, and existing methods to integrate segmentation and timing features.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywilken2022suber

Related papers

• Evaluating Subtitle Segmentation For End-to-end Generation Systems (2022)0.00
• V-SAT: Video Subtitle Annotation Tool (2025)0.00
• Semantic-wer: A Unified Metric For The Evaluation Of ASR Transcript For End Usability (2021)0.00
• Speechbertscore: Reference-aware Automatic Evaluation Of Speech Generation Leveraging NLP Evaluation Metrics (2024)10.74
• Dodging The Data Bottleneck: Automatic Subtitling With Automatically Segmented ST Corpora (2022)2.26
• Direct Speech Translation For Automatic Subtitling (2022)6.77
• Evaluating User Perception Of Speech Recognition System Quality With Semantic Distance Metric (2021)6.77
• Leveraging Broadcast Media Subtitle Transcripts For Automatic Speech Recognition And Subtitling (2025)2.26