Acoustic Modeling For Automatic Lyrics-to-audio Alignment

Abstract

Automatic lyrics to polyphonic audio alignment is a challenging task not only because the vocals are corrupted by background music, but also there is a lack of annotated polyphonic corpus for effective acoustic modeling. In this work, we propose (1) using additional speech and music-informed features and (2) adapting the acoustic models trained on a large amount of solo singing vocals towards polyphonic music using a small amount of in-domain data. Incorporating additional information such as voicing and auditory features together with conventional acoustic features aims to bring robustness against the increased spectro-temporal variations in singing vocals. By adapting the acoustic model using a small amount of polyphonic audio data, we reduce the domain mismatch between training and testing data. We perform several alignment experiments and present an in-depth alignment error analysis on acoustic features, and model adaptation techniques. The results demonstrate that the proposed str

Stats

• citations13
• S2 citations—
• github stars0
• HF likes0
• heat score8.60
• arxiv keygupta2019acoustic

Related papers

• End-to-end Lyrics Alignment For Polyphonic Music Using An Audio-to-character Recognition Model (2019)13.11
• Lyrics-to-audio Alignment By Unsupervised Discovery Of Repetitive Patterns In Vowel Acoustics (2017)6.34
• Adapting Pretrained Speech Model For Mandarin Lyrics Transcription And Alignment (2023)3.58
• A Real-time Lyrics Alignment System Using Chroma And Phonetic Features For Classical Vocal Performance (2024)2.26
• HCLAS-X: Hierarchical And Cascaded Lyrics Alignment System Using Multimodal Cross-correlation (2023)0.00
• Deep Domain Adaptation For Polyphonic Melody Extraction (2022)0.00
• Contrastive Learning-based Audio To Lyrics Alignment For Multiple Languages (2023)6.77
• Content Based Singing Voice Source Separation Via Strong Conditioning Using Aligned Phonemes (2020)0.00