Karaoker: Alignment-free Singing Voice Synthesis With Speech Training Data

Abstract

Existing singing voice synthesis models (SVS) are usually trained on singing data and depend on either error-prone time-alignment and duration features or explicit music score information. In this paper, we propose Karaoker, a multispeaker Tacotron-based model conditioned on voice characteristic features that is trained exclusively on spoken data without requiring time-alignments. Karaoker synthesizes singing voice and transfers style following a multi-dimensional template extracted from a source waveform of an unseen singer/speaker. The model is jointly conditioned with a single deep convolutional encoder on continuous data including pitch, intensity, harmonicity, formants, cepstral peak prominence and octaves. We extend the text-to-speech training objective with feature reconstruction, classification and speaker identification tasks that guide the model to an accurate result. In addition to multitasking, we also employ a Wasserstein GAN training scheme as well as new losses on the ac

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keykakoulidis2022karaoker

Related papers

• Low-resource Cross-domain Singing Voice Synthesis Via Reduced Self-supervised Speech Representations (2024)0.00
• Karasinger: Score-free Singing Voice Synthesis With VQ-VAE Using Mel-spectrograms (2021)2.26
• Adversarially Trained Multi-singer Sequence-to-sequence Singing Synthesizer (2020)7.81
• Karatuner: Towards End To End Natural Pitch Correction For Singing Voice In Karaoke (2021)5.24
• Everyone-can-sing: Zero-shot Singing Voice Synthesis And Conversion With Speech Reference (2025)0.00
• Visinger2+: End-to-end Singing Voice Synthesis Augmented By Self-supervised Learning Representation (2024)4.52
• Cssinger: End-to-end Chunkwise Streaming Singing Voice Synthesis System Based On Conditional Variational Autoencoder (2024)0.00
• Tcsinger: Zero-shot Singing Voice Synthesis With Style Transfer And Multi-level Style Control (2024)7.16