HCLAS-X: Hierarchical And Cascaded Lyrics Alignment System Using Multimodal Cross-correlation

Abstract

In this work, we address the challenge of lyrics alignment, which involves aligning the lyrics and vocal components of songs. This problem requires the alignment of two distinct modalities, namely text and audio. To overcome this challenge, we propose a model that is trained in a supervised manner, utilizing the cross-correlation matrix of latent representations between vocals and lyrics. Our system is designed in a hierarchical and cascaded manner. It predicts synced time first on a sentence-level and subsequently on a word-level. This design enables the system to process long sequences, as the cross-correlation uses quadratic memory with respect to sequence length. In our experiments, we demonstrate that our proposed system achieves a significant improvement in mean average error, showcasing its robustness in comparison to the previous state-of-the-art model. Additionally, we conduct a qualitative analysis of the system after successfully deploying it in several music streaming servi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykang2023hclas

Related papers

• End-to-end Lyrics Alignment For Polyphonic Music Using An Audio-to-character Recognition Model (2019)13.11
• Contrastive Learning-based Audio To Lyrics Alignment For Multiple Languages (2023)6.77
• Deep Cross-modal Correlation Learning For Audio And Lyrics In Music Retrieval (2017)14.06
• Lyrics-to-audio Alignment By Unsupervised Discovery Of Repetitive Patterns In Vowel Acoustics (2017)6.34
• A Real-time Lyrics Alignment System Using Chroma And Phonetic Features For Classical Vocal Performance (2024)2.26
• Acoustic Modeling For Automatic Lyrics-to-audio Alignment (2019)8.60
• Content Based Singing Voice Source Separation Via Strong Conditioning Using Aligned Phonemes (2020)0.00
• Exploiting Synchronized Lyrics And Vocal Features For Music Emotion Detection (2019)0.00