Integrated Multi-level Knowledge Distillation For Enhanced Speaker Verification

Abstract

Knowledge distillation (KD) is widely used in audio tasks, such as speaker verification (SV), by transferring knowledge from a well-trained large model (the teacher) to a smaller, more compact model (the student) for efficiency and portability. Existing KD methods for SV often mirror those used in image processing, focusing on approximating predicted probabilities and hidden representations. However, these methods fail to account for the multi-level temporal properties of speech audio. In this paper, we propose a novel KD method, i.e., Integrated Multi-level Knowledge Distillation (IML-KD), to transfer knowledge of various temporal-scale features of speech from a teacher model to a student model. In the IML-KD, temporal context information from the teacher model is integrated into novel Integrated Gradient-based input-sensitive representations from speech segments with various durations, and the student model is trained to infer these representations with multi-level alignment for the

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyang2024integrated

Related papers

• Emphasized Non-target Speaker Knowledge In Knowledge Distillation For Automatic Speaker Verification (2023)8.35
• Intra-utterance Similarity Preserving Knowledge Distillation For Audio Tagging (2020)3.58
• Distilling Multi-level X-vector Knowledge For Small-footprint Speaker Verification (2023)0.00
• Multi-level Knowledge Distillation For Speech Emotion Recognition In Noisy Conditions (2023)7.81
• VIC-KD: Variance-invariance-covariance Knowledge Distillation To Make Keyword Spotting More Robust Against Adversarial Attacks (2023)2.26
• Inter-kd: Intermediate Knowledge Distillation For Ctc-based Automatic Speech Recognition (2022)7.50
• Predicting Multi-codebook Vector Quantization Indexes For Knowledge Distillation (2022)4.52
• One-step Knowledge Distillation And Fine-tuning In Using Large Pre-trained Self-supervised Learning Models For Speaker Verification (2023)7.81