Songglm: Lyric-to-melody Generation With 2D Alignment Encoding And Multi-task Pre-training

Abstract

Lyric-to-melody generation aims to automatically create melodies based on given lyrics, requiring the capture of complex and subtle correlations between them. However, previous works usually suffer from two main challenges: 1) lyric-melody alignment modeling, which is often simplified to one-syllable/word-to-one-note alignment, while others have the problem of low alignment accuracy; 2) lyric-melody harmony modeling, which usually relies heavily on intermediates or strict rules, limiting model's capabilities and generative diversity. In this paper, we propose SongGLM, a lyric-to-melody generation system that leverages 2D alignment encoding and multi-task pre-training based on the General Language Model (GLM) to guarantee the alignment and harmony between lyrics and melodies. Specifically, 1) we introduce a unified symbolic song representation for lyrics and melodies with word-level and phrase-level (2D) alignment encoding to capture the lyric-melody alignment; 2) we design a multi-task

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyyu2024songglm

Related papers

• Conditional LSTM-GAN For Melody Generation From Lyrics (2019)14.69
• Unsupervised Melody-to-lyric Generation (2023)0.00
• Joint Learning Of Wording And Formatting For Singable Melody-to-lyric Generation (2023)0.00
• Songmass: Automatic Song Writing With Pre-training And Alignment Constraint (2020)11.39
• CSL-L2M: Controllable Song-level Lyric-to-melody Generation Based On Conditional Transformer With Fine-grained Lyric And Musical Controls (2024)2.26
• Interpretable Melody Generation From Lyrics With Discrete-valued Adversarial Training (2022)6.34
• Diverse Melody Generation From Chinese Lyrics Via Mutual Information Maximization (2020)0.00
• A Syllable-structured, Contextually-based Conditionally Generation Of Chinese Lyrics (2019)7.16