Speech Enhancement And Dereverberation With Diffusion-based Generative Models

Abstract

In this work, we build upon our previous publication and use diffusion-based generative models for speech enhancement. We present a detailed overview of the diffusion process that is based on a stochastic differential equation and delve into an extensive theoretical examination of its implications. Opposed to usual conditional generation tasks, we do not start the reverse process from pure Gaussian noise but from a mixture of noisy speech and Gaussian noise. This matches our forward process which moves from clean speech to noisy speech by including a drift term. We show that this procedure enables using only 30 diffusion steps to generate high-quality clean speech estimates. By adapting the network architecture, we are able to significantly improve the speech enhancement performance, indicating that the network, rather than the formalism, was the main limitation of our original approach. In an extensive cross-dataset evaluation, we show that the improved method can compete with recent

Stats

• citations233
• S2 citations—
• github stars743
• HF likes0
• heat score23.51
• arxiv keyrichter2022speech

Related papers

• Storm: A Diffusion-based Stochastic Regeneration Model For Speech Enhancement And Dereverberation (2022)15.43
• Single And Few-step Diffusion For Generative Speech Enhancement (2023)10.21
• Investigating The Design Space Of Diffusion Models For Speech Enhancement (2023)10.07
• Analysing Diffusion-based Generative Approaches Versus Discriminative Approaches For Speech Restoration (2022)11.39
• Diffusion-based Speech Enhancement With A Weighted Generative-supervised Learning Loss (2023)0.00
• Extract And Diffuse: Latent Integration For Improved Diffusion-based Speech And Vocal Enhancement (2024)0.00
• Diffusion-based Signal Refiner For Speech Enhancement And Separation (2023)2.26
• Diffusion-based Speech Enhancement With Schr\"odinger Bridge And Symmetric Noise Schedule (2024)0.00