A Real-time Lyrics Alignment System Using Chroma And Phonetic Features For Classical Vocal Performance

Abstract

The goal of real-time lyrics alignment is to take live singing audio as input and to pinpoint the exact position within given lyrics on the fly. The task can benefit real-world applications such as the automatic subtitling of live concerts or operas. However, designing a real-time model poses a great challenge due to the constraints of only using past input and operating within a minimal latency. Furthermore, due to the lack of datasets for real-time models for lyrics alignment, previous studies have mostly evaluated with private in-house datasets, resulting in a lack of standard evaluation methods. This paper presents a real-time lyrics alignment system for classical vocal performances with two contributions. First, we improve the lyrics alignment algorithm by finding an optimal combination of chromagram and phonetic posteriorgram (PPG) that capture melodic and phonetics features of the singing voice, respectively. Second, we recast the Schubert Winterreise Dataset (SWD) which contain

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keypark2024a

Related papers

• End-to-end Lyrics Alignment For Polyphonic Music Using An Audio-to-character Recognition Model (2019)13.11
• HCLAS-X: Hierarchical And Cascaded Lyrics Alignment System Using Multimodal Cross-correlation (2023)0.00
• Acoustic Modeling For Automatic Lyrics-to-audio Alignment (2019)8.60
• Lyrics-to-audio Alignment By Unsupervised Discovery Of Repetitive Patterns In Vowel Acoustics (2017)6.34
• Contrastive Learning-based Audio To Lyrics Alignment For Multiple Languages (2023)6.77
• Adapting Pretrained Speech Model For Mandarin Lyrics Transcription And Alignment (2023)3.58
• Content Based Singing Voice Source Separation Via Strong Conditioning Using Aligned Phonemes (2020)0.00
• Songglm: Lyric-to-melody Generation With 2D Alignment Encoding And Multi-task Pre-training (2024)3.58