On The Transferability Of Whisper-based Representations For "in-the-wild" Cross-task Downstream Speech Applications

Abstract

Large self-supervised pre-trained speech models have achieved remarkable success across various speech-processing tasks. The self-supervised training of these models leads to universal speech representations that can be used for different downstream tasks, ranging from automatic speech recognition (ASR) to speaker identification. Recently, Whisper, a transformer-based model was proposed and trained on large amount of weakly supervised data for ASR; it outperformed several state-of-the-art self-supervised models. Given the superiority of Whisper for ASR, in this paper we explore the transferability of the representation for four other speech tasks in SUPERB benchmark. Moreover, we explore the robustness of Whisper representation for ``in the wild'' tasks where speech is corrupted by environment noise and room reverberation. Experimental results show Whisper achieves promising results across tasks and environmental conditions, thus showing potential for cross-task real-world deployment.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychemudupati2023on

Related papers

• An Exploration Into The Performance Of Unsupervised Cross-task Speech Representations For "in The Wild'' Edge Applications (2023)0.00
• Word-level Embeddings For Cross-task Transfer Learning In Speech Processing (2019)5.24
• An Exploration Of Self-supervised Pretrained Representations For End-to-end Speech Recognition (2021)12.25
• Multilingual Distilwhisper: Efficient Distillation Of Multi-task Speech Models Via Language-specific Experts (2023)8.09
• Cross-lingual Transfer Learning For Speech Translation (2024)6.34
• Investigating Zero-shot Generalizability On Mandarin-english Code-switched ASR And Speech-to-text Translation Of Recent Foundation Models With Self-supervision And Weak Supervision (2023)0.00
• Whisper Speaker Identification: Leveraging Pre-trained Multilingual Transformers For Robust Speaker Embeddings (2025)0.00
• End-to-end Whisper To Natural Speech Conversion Using Modified Transformer Network (2020)0.00