Accurate And Structured Pruning For Efficient Automatic Speech Recognition

Abstract

Automatic Speech Recognition (ASR) has seen remarkable advancements with deep neural networks, such as Transformer and Conformer. However, these models typically have large model sizes and high inference costs, posing a challenge to deploy on resource-limited devices. In this paper, we propose a novel compression strategy that leverages structured pruning and knowledge distillation to reduce the model size and inference cost of the Conformer model while preserving high recognition performance. Our approach utilizes a set of binary masks to indicate whether to retain or prune each Conformer module, and employs L0 regularization to learn the optimal mask values. To further enhance pruning performance, we use a layerwise distillation strategy to transfer knowledge from unpruned to pruned models. Our method outperforms all pruning baselines on the widely used LibriSpeech benchmark, achieving a 50% reduction in model size and a 28% reduction in inference cost with minimal performance loss.

Stats

• citations10
• S2 citations—
• github stars0
• HF likes0
• heat score7.81
• arxiv keyjiang2023accurate

Related papers

• Structured Pruning Of Self-supervised Pre-trained Models For Speech Recognition And Understanding (2023)11.39
• Dynamic Encoder Size Based On Data-driven Layer-wise Pruning For Speech Recognition (2024)5.24
• Dynamic Sparsity Neural Networks For Automatic Speech Recognition (2020)0.00
• Synergistic Effects Of Knowledge Distillation And Structured Pruning For Self-supervised Speech Models (2025)0.00
• Federated Pruning: Improving Neural Network Efficiency With Federated Learning (2022)7.50
• Efficient Conformer: Progressive Downsampling And Grouped Attention For Automatic Speech Recognition (2021)13.79
• Efficientasr: Speech Recognition Network Compression Via Attention Redundancy And Chunk-level FFN Optimization (2024)3.58
• Speechprune: Context-aware Token Pruning For Speech Information Retrieval (2024)3.58