Extending Whisper With Prompt Tuning To Target-speaker ASR

Abstract

Target-speaker automatic speech recognition (ASR) aims to transcribe the desired speech of a target speaker from multi-talker overlapped utterances. Most of the existing target-speaker ASR (TS-ASR) methods involve either training from scratch or fully fine-tuning a pre-trained model, leading to significant training costs and becoming inapplicable to large foundation models. This work leverages prompt tuning, a parameter-efficient fine-tuning approach, to extend Whisper, a large-scale single-talker ASR model, to TS-ASR. Variants of prompt tuning approaches along with their configurations are explored and optimized for TS-ASR.Experimental results show that prompt tuning can achieve performance comparable to state-of-the-art full training approaches while only requiring about 1% of task-specific model parameters. Notably, the original Whisper's features, such as inverse text normalization and timestamp tagging, are retained in target-speaker ASR, keeping the generated transcriptions natur

Stats

• citations18
• S2 citations—
• github stars0
• HF likes0
• heat score9.59
• arxiv keyma2023extending

Related papers

• Target Speaker ASR With Whisper (2024)7.16
• Prompting The Hidden Talent Of Web-scale Speech Models For Zero-shot Task Generalization (2023)16.38
• Multilingual Distilwhisper: Efficient Distillation Of Multi-task Speech Models Via Language-specific Experts (2023)8.09
• Whisper-lm: Improving ASR Models With Language Models For Low-resource Languages (2025)3.29
• Simul-whisper: Attention-guided Streaming Whisper With Truncation Detection (2024)6.34
• M2r-whisper: Multi-stage And Multi-scale Retrieval Augmentation For Enhancing Whisper (2024)6.77
• Contextual Biasing To Improve Domain-specific Custom Vocabulary Audio Transcription Without Explicit Fine-tuning Of Whisper Model (2024)4.52
• Probing The Hidden Talent Of ASR Foundation Models For L2 English Oral Assessment (2025)0.00