Leveraging Whisper Embeddings For Audio-based Lyrics Matching

Abstract

Audio-based lyrics matching can be an appealing alternative to other content-based retrieval approaches, but existing methods often suffer from limited reproducibility and inconsistent baselines. In this work, we introduce WEALY, a fully reproducible pipeline that leverages Whisper decoder embeddings for lyrics matching tasks. WEALY establishes robust and transparent baselines, while also exploring multimodal extensions that integrate textual and acoustic features. Through extensive experiments on standard datasets, we demonstrate that WEALY achieves a performance comparable to state-of-the-art methods that lack reproducibility. In addition, we provide ablation studies and analyses on language robustness, loss functions, and embedding strategies. This work contributes a reliable benchmark for future research, and underscores the potential of speech technologies for music information retrieval tasks.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymancini2025leveraging

Related papers

• Adapting Pretrained Speech Model For Mandarin Lyrics Transcription And Alignment (2023)3.58
• Liwhiz: A Non-intrusive Lyric Intelligibility Prediction System For The Cadenza Challenge (2025)2.26
• Whisper Speaker Identification: Leveraging Pre-trained Multilingual Transformers For Robust Speaker Embeddings (2025)0.00
• Exploiting Synchronized Lyrics And Vocal Features For Music Emotion Detection (2019)0.00
• Lyricwhiz: Robust Multilingual Zero-shot Lyrics Transcription By Whispering To Chatgpt (2023)0.00
• End-to-end Lyrics Alignment For Polyphonic Music Using An Audio-to-character Recognition Model (2019)13.11
• Leveraging Multilingual Transfer For Unsupervised Semantic Acoustic Word Embeddings (2023)3.58
• Contrastive Learning-based Audio To Lyrics Alignment For Multiple Languages (2023)6.77