Improved Training For End-to-end Streaming Automatic Speech Recognition Model With Punctuation

Abstract

Punctuated text prediction is crucial for automatic speech recognition as it enhances readability and impacts downstream natural language processing tasks. In streaming scenarios, the ability to predict punctuation in real-time is particularly desirable but presents a difficult technical challenge. In this work, we propose a method for predicting punctuated text from input speech using a chunk-based Transformer encoder trained with Connectionist Temporal Classification (CTC) loss. The acoustic model trained with long sequences by concatenating the input and target sequences can learn punctuation marks attached to the end of sentences more effectively. Additionally, by combining CTC losses on the chunks and utterances, we achieved both the improved F1 score of punctuation prediction and Word Error Rate (WER).

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykim2023improved

Related papers

• Streaming Punctuation: A Novel Punctuation Technique Leveraging Bidirectional Context For Continuous Speech Recognition (2023)4.52
• A Light-weight And Efficient Punctuation And Word Casing Prediction Model For On-device Streaming ASR (2024)0.00
• Longer Is (not Necessarily) Stronger: Punctuated Long-sequence Training For Enhanced Speech Recognition And Translation (2024)4.52
• End To End ASR System With Automatic Punctuation Insertion (2020)0.00
• Controllable Time-delay Transformer For Real-time Punctuation Prediction And Disfluency Detection (2020)10.48
• Efficient Ensemble For Multimodal Punctuation Restoration Using Time-delay Neural Network (2023)1.91
• Replacing Human Audio With Synthetic Audio For On-device Unspoken Punctuation Prediction (2020)0.00
• Multimodal Semi-supervised Learning Framework For Punctuation Prediction In Conversational Speech (2020)9.59