Dualvc 2: Dynamic Masked Convolution For Unified Streaming And Non-streaming Voice Conversion

Abstract

Voice conversion is becoming increasingly popular, and a growing number of application scenarios require models with streaming inference capabilities. The recently proposed DualVC attempts to achieve this objective through streaming model architecture design and intra-model knowledge distillation along with hybrid predictive coding to compensate for the lack of future information. However, DualVC encounters several problems that limit its performance. First, the autoregressive decoder has error accumulation in its nature and limits the inference speed as well. Second, the causal convolution enables streaming capability but cannot sufficiently use future information within chunks. Third, the model is unable to effectively address the noise in the unvoiced segments, lowering the sound quality. In this paper, we propose DualVC 2 to address these issues. Specifically, the model backbone is migrated to a Conformer-based architecture, empowering parallel inference. Causal convolution is repl

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyning2023dualvc

Related papers

• Dualvc 3: Leveraging Language Model Generated Pseudo Context For End-to-end Low Latency Streaming Voice Conversion (2024)0.00
• Fastvc: Fast Voice Conversion With Non-parallel Data (2020)5.24
• Discrete Unit Based Masking For Improving Disentanglement In Voice Conversion (2024)0.00
• ACVAE-VC: Non-parallel Many-to-many Voice Conversion With Auxiliary Classifier Variational Autoencoder (2018)14.69
• Dynamic Chunk Convolution For Unified Streaming And Non-streaming Conformer ASR (2023)6.77
• Convs2s-vc: Fully Convolutional Sequence-to-sequence Voice Conversion (2018)12.68
• Converting Anyone's Voice: End-to-end Expressive Voice Conversion With A Conditional Diffusion Model (2024)5.24
• Stepback: Enhanced Disentanglement For Voice Conversion Via Multi-task Learning (2025)0.00