Diffusion-based Speech Enhancement With Joint Generative And Predictive Decoders

Abstract

Diffusion-based generative speech enhancement (SE) has recently received attention, but reverse diffusion remains time-consuming. One solution is to initialize the reverse diffusion process with enhanced features estimated by a predictive SE system. However, the pipeline structure currently does not consider for a combined use of generative and predictive decoders. The predictive decoder allows us to use the further complementarity between predictive and diffusion-based generative SE. In this paper, we propose a unified system that use jointly generative and predictive decoders across two levels. The encoder encodes both generative and predictive information at the shared encoding level. At the decoded feature level, we fuse the two decoded features by generative and predictive decoders. Specifically, the two SE modules are fused in the initial and final diffusion steps: the initial fusion initializes the diffusion process with the predictive SE to improve convergence, and the final fu

Stats

• citations18
• S2 citations—
• github stars0
• HF likes0
• heat score9.59
• arxiv keyshi2023diffusion

Related papers

• Single And Few-step Diffusion For Generative Speech Enhancement (2023)10.21
• Diffusion-based Speech Enhancement With A Weighted Generative-supervised Learning Loss (2023)0.00
• Speech Enhancement And Dereverberation With Diffusion-based Generative Models (2022)23.51
• Diffusion-based Generative Modeling With Discriminative Guidance For Streamable Speech Enhancement (2024)7.16
• Noise-aware Speech Enhancement Using Diffusion Probabilistic Model (2023)8.82
• Storm: A Diffusion-based Stochastic Regeneration Model For Speech Enhancement And Dereverberation (2022)15.43
• Extract And Diffuse: Latent Integration For Improved Diffusion-based Speech And Vocal Enhancement (2024)0.00
• Diffusion-based Speech Enhancement With Schr\"odinger Bridge And Symmetric Noise Schedule (2024)0.00