Fast Conformer With Linearly Scalable Attention For Efficient Speech Recognition

Abstract

Conformer-based models have become the dominant end-to-end architecture for speech processing tasks. With the objective of enhancing the conformer architecture for efficient training and inference, we carefully redesigned Conformer with a novel downsampling schema. The proposed model, named Fast Conformer(FC), is 2.8x faster than the original Conformer, supports scaling to Billion parameters without any changes to the core architecture and also achieves state-of-the-art accuracy on Automatic Speech Recognition benchmarks. To enable transcription of long-form speech up to 11 hours, we replaced global attention with limited context attention post-training, while also improving accuracy through fine-tuning with the addition of a global token. Fast Conformer, when combined with a Transformer decoder also outperforms the original Conformer in accuracy and in speed for Speech Translation and Spoken Language Understanding.

Stats

• citations84
• S2 citations—
• github stars0
• HF likes0
• heat score14.47
• arxiv keyrekesh2023fast

Related papers

• Efficient Conformer: Progressive Downsampling And Grouped Attention For Automatic Speech Recognition (2021)13.79
• Efficient Conformer With Prob-sparse Attention Mechanism For End-to-endspeech Recognition (2021)8.09
• Nextformer: A Convnext Augmented Conformer For End-to-end Speech Recognition (2022)0.00
• Stateful Conformer With Cache-based Inference For Streaming Automatic Speech Recognition (2023)8.60
• Efficient And Robust Long-form Speech Recognition With Hybrid H3-conformer (2024)3.58
• PCNN: A Lightweight Parallel Conformer Neural Network For Efficient Monaural Speech Enhancement (2023)6.77
• Df-conformer: Integrated Architecture Of Conv-tasnet And Conformer Using Linear Complexity Self-attention For Speech Enhancement (2021)11.29
• Key Frame Mechanism For Efficient Conformer Based End-to-end Speech Recognition (2023)3.58