Conditional End-to-end Audio Transforms

Abstract

We present an end-to-end method for transforming audio from one style to another. For the case of speech, by conditioning on speaker identities, we can train a single model to transform words spoken by multiple people into multiple target voices. For the case of music, we can specify musical instruments and achieve the same result. Architecturally, our method is a fully-differentiable sequence-to-sequence model based on convolutional and hierarchical recurrent neural networks. It is designed to capture long-term acoustic dependencies, requires minimal post-processing, and produces realistic audio transforms. Ablation studies confirm that our model can separate speaker and instrument properties from acoustic content at different receptive fields. Empirically, our method achieves competitive performance on community-standard datasets.

Stats

• citations14
• S2 citations—
• github stars0
• HF likes0
• heat score8.82
• arxiv keyhaque2018conditional

Related papers

• Differentiable WORLD Synthesizer-based Neural Vocoder With Application To End-to-end Audio Style Transfer (2022)0.00
• Blow: A Single-scale Hyperconditioned Flow For Non-parallel Raw-audio Voice Conversion (2019)0.00
• Voice Conversion With Conditional Samplernn (2018)7.50
• Time Domain Neural Audio Style Transfer (2017)0.00
• Utilizing Domain Knowledge In End-to-end Audio Processing (2017)0.00
• Speaker Conditioning Of Acoustic Models Using Affine Transformation For Multi-speaker Speech Recognition (2021)0.00
• Audiovisual Speaker Conversion: Jointly And Simultaneously Transforming Facial Expression And Acoustic Characteristics (2018)4.52
• Spectron: Target Speaker Extraction Using Conditional Transformer With Adversarial Refinement (2024)0.00