Semi-supervised Learning For Singing Synthesis Timbre

Abstract

We propose a semi-supervised singing synthesizer, which is able to learn new voices from audio data only, without any annotations such as phonetic segmentation. Our system is an encoder-decoder model with two encoders, linguistic and acoustic, and one (acoustic) decoder. In a first step, the system is trained in a supervised manner, using a labelled multi-singer dataset. Here, we ensure that the embeddings produced by both encoders are similar, so that we can later use the model with either acoustic or linguistic input features. To learn a new voice in an unsupervised manner, the pretrained acoustic encoder is used to train a decoder for the target singer. Finally, at inference, the pretrained linguistic encoder is used together with the decoder of the new voice, to produce acoustic features from linguistic input. We evaluate our system with a listening test and show that the results are comparable to those obtained with an equivalent supervised approach.

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keybonada2020semi

Related papers

• Deep Audio-visual Singing Voice Transcription Based On Self-supervised Learning Models (2023)0.00
• A Melody-unsupervision Model For Singing Voice Synthesis (2021)5.84
• Adversarial Multi-task Learning For Disentangling Timbre And Pitch In Singing Voice Synthesis (2022)4.52
• Adversarially Trained Multi-singer Sequence-to-sequence Singing Synthesizer (2020)7.81
• Visinger2+: End-to-end Singing Voice Synthesis Augmented By Self-supervised Learning Representation (2024)4.52
• Enhancing The Vocal Range Of Single-speaker Singing Voice Synthesis With Melody-unsupervised Pre-training (2023)3.58
• Learn2sing: Target Speaker Singing Voice Synthesis By Learning From A Singing Teacher (2020)4.52
• A Preliminary Investigation On Flexible Singing Voice Synthesis Through Decomposed Framework With Inferrable Features (2024)0.00