Transformers With Convolutional Context For ASR

Abstract

The recent success of transformer networks for neural machine translation and other NLP tasks has led to a surge in research work trying to apply it for speech recognition. Recent efforts studied key research questions around ways of combining positional embedding with speech features, and stability of optimization for large scale learning of transformer networks. In this paper, we propose replacing the sinusoidal positional embedding for transformers with convolutionally learned input representations. These contextual representations provide subsequent transformer blocks with relative positional information needed for discovering long-range relationships between local concepts. The proposed system has favorable optimization characteristics where our reported results are produced with fixed learning rate of 1.0 and no warmup steps. The proposed model achieves a competitive 4.7% and 12.9% WER on the Librispeech ``test clean'' and ``test other'' subsets when no extra LM text is provided.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymohamed2019transformers

Related papers

• Towards Effective And Compact Contextual Representation For Conformer Transducer Speech Recognition Systems (2023)7.16
• Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers (2021)10.07
• Transformer ASR With Contextual Block Processing (2019)0.00
• Improving Transformer-based Conversational ASR By Inter-sentential Attention Mechanism (2022)7.50
• Transformer-based Acoustic Modeling For Hybrid Speech Recognition (2019)16.30
• Contextnet: Improving Convolutional Neural Networks For Automatic Speech Recognition With Global Context (2020)17.24
• Transformer-transducer: End-to-end Speech Recognition With Self-attention (2019)0.00
• Conv-transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-end Speech Recognition (2020)11.08