Language-queried Target Sound Extraction Without Parallel Training Data

Abstract

Language-queried target sound extraction (TSE) aims to extract specific sounds from mixtures based on language queries. Traditional fully-supervised training schemes require extensively annotated parallel audio-text data, which are labor-intensive. We introduce a parallel-data-free training scheme, requiring only unlabelled audio clips for TSE model training by utilizing the contrastive language-audio pre-trained model (CLAP). In a vanilla parallel-data-free training stage, target audio is encoded using the pre-trained CLAP audio encoder to form a condition embedding, while during testing, user language queries are encoded by CLAP text encoder as the condition embedding. This vanilla approach assumes perfect alignment between text and audio embeddings, which is unrealistic. Two major challenges arise from training-testing mismatch: the persistent modality gap between text and audio and the risk of overfitting due to the exposure of rich acoustic details in target audio embedding during

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keyma2024language

Related papers

• Soloaudio: Target Sound Extraction With Language-oriented Audio Diffusion Transformer (2024)7.50
• Improving Curriculum Learning For Target Speaker Extraction With Synthetic Speakers (2024)2.26
• Clapspeech: Learning Prosody From Text Context With Contrastive Language-audio Pre-training (2023)0.00
• Typing To Listen At The Cocktail Party: Text-guided Target Speaker Extraction (2023)3.58
• Clipsonic: Text-to-audio Synthesis With Unlabeled Videos And Pretrained Language-vision Models (2023)9.03
• Do Audio-language Models Understand Linguistic Variations? (2024)0.00
• MATS: An Audio Language Model Under Text-only Supervision (2025)0.00
• CLASP: Contrastive Language-speech Pretraining For Multilingual Multimodal Information Retrieval (2024)0.00