End-to-end Lyrics Alignment For Polyphonic Music Using An Audio-to-character Recognition Model

Abstract

Time-aligned lyrics can enrich the music listening experience by enabling karaoke, text-based song retrieval and intra-song navigation, and other applications. Compared to text-to-speech alignment, lyrics alignment remains highly challenging, despite many attempts to combine numerous sub-modules including vocal separation and detection in an effort to break down the problem. Furthermore, training required fine-grained annotations to be available in some form. Here, we present a novel system based on a modified Wave-U-Net architecture, which predicts character probabilities directly from raw audio using learnt multi-scale representations of the various signal components. There are no sub-modules whose interdependencies need to be optimized. Our training procedure is designed to work with weak, line-level annotations available in the real world. With a mean alignment error of 0.35s on a standard dataset our system outperforms the state-of-the-art by an order of magnitude.

Stats

• citations55
• S2 citations—
• github stars0
• HF likes0
• heat score13.11
• arxiv keystoller2019end

Related papers

• Acoustic Modeling For Automatic Lyrics-to-audio Alignment (2019)8.60
• Lyrics-to-audio Alignment By Unsupervised Discovery Of Repetitive Patterns In Vowel Acoustics (2017)6.34
• Contrastive Learning-based Audio To Lyrics Alignment For Multiple Languages (2023)6.77
• HCLAS-X: Hierarchical And Cascaded Lyrics Alignment System Using Multimodal Cross-correlation (2023)0.00
• A Real-time Lyrics Alignment System Using Chroma And Phonetic Features For Classical Vocal Performance (2024)2.26
• Content Based Singing Voice Source Separation Via Strong Conditioning Using Aligned Phonemes (2020)0.00
• Adapting Pretrained Speech Model For Mandarin Lyrics Transcription And Alignment (2023)3.58
• Songtrans: An Unified Song Transcription And Alignment Method For Lyrics And Notes (2024)0.00