Self-attention Networks For Connectionist Temporal Classification In Speech Recognition

Abstract

The success of self-attention in NLP has led to recent applications in end-to-end encoder-decoder architectures for speech recognition. Separately, connectionist temporal classification (CTC) has matured as an alignment-free, non-autoregressive approach to sequence transduction, either by itself or in various multitask and decoding frameworks. We propose SAN-CTC, a deep, fully self-attentional network for CTC, and show it is tractable and competitive for end-to-end speech recognition. SAN-CTC trains quickly and outperforms existing CTC models and most encoder-decoder models, with character error rates (CERs) of 4.7% in 1 day on WSJ eval92 and 2.8% in 1 week on LibriSpeech test-clean, with a fixed architecture and one GPU. Similar improvements hold for WERs after LM decoding. We motivate the architecture for speech, evaluate position and downsampling approaches, and explore how label alphabets (character, phoneme, subword) affect attention heads and performance.

Stats

• citations86
• S2 citations—
• github stars0
• HF likes0
• heat score14.55
• arxiv keysalazar2019self

Related papers

• Advancing Connectionist Temporal Classification With Attention Modeling (2018)11.49
• An Improved Hybrid Ctc-attention Model For Speech Recognition (2018)0.00
• A Study Of All-convolutional Encoders For Connectionist Temporal Classification (2017)5.84
• Joint Ctc-attention Based End-to-end Speech Recognition Using Multi-task Learning (2016)20.43
• BERT Meets CTC: New Formulation Of End-to-end Speech Recognition With Pre-trained Masked Language Model (2022)0.00
• Residual Convolutional CTC Networks For Automatic Speech Recognition (2017)0.00
• Advances In Joint Ctc-attention Based End-to-end Speech Recognition With A Deep CNN Encoder And RNN-LM (2017)16.49
• Speech Enhancement Using Multi-stage Self-attentive Temporal Convolutional Networks (2021)14.15