Structured Pruning Of Self-supervised Pre-trained Models For Speech Recognition And Understanding
2023 · Yifan Peng, Kwangyoun Kim, Felix Wu, et al.
Abstract
Self-supervised speech representation learning (SSL) has shown to be effective in various downstream tasks, but SSL models are usually large and slow. Model compression techniques such as pruning aim to reduce the model size and computation without degradation in accuracy. Prior studies focus on the pruning of Transformers; however, speech models not only utilize a stack of Transformer blocks, but also combine a frontend network based on multiple convolutional layers for low-level feature representation learning. This frontend has a small size but a heavy computational cost. In this work, we propose three task-specific structured pruning methods to deal with such heterogeneous networks. Experiments on LibriSpeech and SLURP show that the proposed method is more accurate than the original wav2vec2-base with 10% to 30% less computation, and is able to reduce the computation by 40% to 50% without any degradation.
Authors
(none)
Tags
Stats
Related papers
- Accurate And Structured Pruning For Efficient Automatic Speech Recognition (2023)7.81
- Synergistic Effects Of Knowledge Distillation And Structured Pruning For Self-supervised Speech Models (2025)0.00
- Distillation And Pruning For Scalable Self-supervised Representation-based Speech Quality Assessment (2025)8.09
- Fine-tuning Strategies For Faster Inference Using Speech Self-supervised Models: A Comparative Study (2023)8.35
- An Adapter Based Pre-training For Efficient And Scalable Self-supervised Speech Representation Learning (2021)8.35
- Dynamic Encoder Size Based On Data-driven Layer-wise Pruning For Speech Recognition (2024)5.24
- Recycle-and-distill: Universal Compression Strategy For Transformer-based Speech SSL Models With Attention Map Reusing And Masking Distillation (2023)5.84
- Progressive Residual Extraction Based Pre-training For Speech Representation Learning (2024)0.00