Efficient Conformer: Progressive Downsampling And Grouped Attention For Automatic Speech Recognition

Abstract

The recently proposed Conformer architecture has shown state-of-the-art performances in Automatic Speech Recognition by combining convolution with attention to model both local and global dependencies. In this paper, we study how to reduce the Conformer architecture complexity with a limited computing budget, leading to a more efficient architecture design that we call Efficient Conformer. We introduce progressive downsampling to the Conformer encoder and propose a novel attention mechanism named grouped attention, allowing us to reduce attention complexity from \(O(n^\{2\}d)\) to \(O(n^\{2\}d / g)\) for sequence length \(n\), hidden dimension \(d\) and group size parameter \(g\). We also experiment the use of strided multi-head self-attention as a global downsampling operation. Our experiments are performed on the LibriSpeech dataset with CTC and RNN-Transducer losses. We show that within the same computing budget, the proposed architecture achieves better performances with faster tra

Stats

• citations68
• S2 citations—
• github stars0
• HF likes0
• heat score13.79
• arxiv keyburchi2021efficient

Related papers

• Fast Conformer With Linearly Scalable Attention For Efficient Speech Recognition (2023)14.47
• Efficient Conformer With Prob-sparse Attention Mechanism For End-to-endspeech Recognition (2021)8.09
• Adding Connectionist Temporal Summarization Into Conformer To Improve Its Decoder Efficiency For Speech Recognition (2022)0.00
• Df-conformer: Integrated Architecture Of Conv-tasnet And Conformer Using Linear Complexity Self-attention For Speech Enhancement (2021)11.29
• Nextformer: A Convnext Augmented Conformer For End-to-end Speech Recognition (2022)0.00
• Efficient And Robust Long-form Speech Recognition With Hybrid H3-conformer (2024)3.58
• Stateful Conformer With Cache-based Inference For Streaming Automatic Speech Recognition (2023)8.60
• Accurate And Structured Pruning For Efficient Automatic Speech Recognition (2023)7.81