Synergistic Effects Of Knowledge Distillation And Structured Pruning For Self-supervised Speech Models

Abstract

Traditionally, Knowledge Distillation (KD) is used for model compression, often leading to suboptimal performance. In this paper, we evaluate the impact of combining KD loss with alternative pruning techniques, including Low-Rank Factorization (LRF) and l0 regularization, on a conformer-based pre-trained network under the paradigm of Self-Supervised Learning (SSL). We also propose a strategy to jointly prune and train an RNN-T-based ASR model, demonstrating that this approach yields superior performance compared to pruning a pre-trained network first and then using it for ASR training. This approach led to a significant reduction in word error rate: l0 and KD combination achieves the best non-streaming performance, with a 8.9% Relative Word Error Rate (RWER) improvement over the baseline, while LRF and KD combination yields the best results for streaming ASR, improving RWER by 13.4%.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyc2025synergistic

Related papers

• Structured Pruning Of Self-supervised Pre-trained Models For Speech Recognition And Understanding (2023)11.39
• Deep Versus Wide: An Analysis Of Student Architectures For Task-agnostic Knowledge Distillation Of Self-supervised Speech Models (2022)9.23
• SKILL: Similarity-aware Knowledge Distillation For Speech Self-supervised Learning (2024)3.58
• Accurate And Structured Pruning For Efficient Automatic Speech Recognition (2023)7.81
• Fine-tuning Strategies For Faster Inference Using Speech Self-supervised Models: A Comparative Study (2023)8.35
• One-step Knowledge Distillation And Fine-tuning In Using Large Pre-trained Self-supervised Learning Models For Speaker Verification (2023)7.81
• Application Of Knowledge Distillation To Multi-task Speech Representation Learning (2022)2.26
• PADA: Pruning Assisted Domain Adaptation For Self-supervised Speech Representations (2022)5.24