Noise Robust Distillation Of Self-supervised Speech Models Via Correlation Metrics

Abstract

Compared to large speech foundation models, small distilled models exhibit degraded noise robustness. The student's robustness can be improved by introducing noise at the inputs during pre-training. Despite this, using the standard distillation loss still yields a student with degraded performance. Thus, this paper proposes improving student robustness via distillation with correlation metrics. Teacher behavior is learned by maximizing the teacher and student cross-correlation matrix between their representations towards identity. Noise robustness is encouraged via the student's self-correlation minimization. The proposed method is agnostic of the teacher model and consistently outperforms the previous approach. This work also proposes an heuristic to weigh the importance of the two correlation terms automatically. Experiments show consistently better clean and noise generalization on Intent Classification, Keyword Spotting, and Automatic Speech Recognition tasks on SUPERB Challenge.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyrittergutierrez2023noise

Related papers

• An Efficient End-to-end Approach To Noise Invariant Speech Features Via Multi-task Learning (2024)0.00
• Distilling Knowledge From Ensembles Of Acoustic Models For Joint Ctc-attention End-to-end Speech Recognition (2020)8.09
• Star: Distilling Speech Temporal Relation For Lightweight Speech Self-supervised Learning Models (2023)5.24
• Distillation And Pruning For Scalable Self-supervised Representation-based Speech Quality Assessment (2025)8.09
• Codert: Distilling Encoder Representations With Co-learning For Transducer-based Speech Recognition (2021)6.77
• Gendistiller: Distilling Pre-trained Language Models Based On An Autoregressive Generative Model (2024)2.26
• VIC-KD: Variance-invariance-covariance Knowledge Distillation To Make Keyword Spotting More Robust Against Adversarial Attacks (2023)2.26
• Dual-branch Knowledge Distillation For Noise-robust Synthetic Speech Detection (2023)9.07