Diffnorm: Self-supervised Normalization For Non-autoregressive Speech-to-speech Translation

Abstract

Non-autoregressive Transformers (NATs) are recently applied in direct speech-to-speech translation systems, which convert speech across different languages without intermediate text data. Although NATs generate high-quality outputs and offer faster inference than autoregressive models, they tend to produce incoherent and repetitive results due to complex data distribution (e.g., acoustic and linguistic variations in speech). In this work, we introduce DiffNorm, a diffusion-based normalization strategy that simplifies data distributions for training NAT models. After training with a self-supervised noise estimation objective, DiffNorm constructs normalized target data by denoising synthetically corrupted speech features. Additionally, we propose to regularize NATs with classifier-free guidance, improving model robustness and translation quality by randomly dropping out source information during training. Our strategies result in a notable improvement of about +7 ASR-BLEU for English-Spa

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytan2024diffnorm

Related papers

• Feature Normalization For Fine-tuning Self-supervised Models In Speech Enhancement (2023)2.26
• Improving Robustness Of Neural Inverse Text Normalization Via Data-augmentation, Semi-supervised Learning, And Post-aligning Method (2023)0.00
• Diffspeaker: Speech-driven 3D Facial Animation With Diffusion Transformer (2024)5.24
• Norespeech: Knowledge Distillation Based Conditional Diffusion Model For Noise-robust Expressive TTS (2022)0.00
• Robust Neural Machine Translation For Clean And Noisy Speech Transcripts (2019)0.00
• Naturalspeech 2: Latent Diffusion Models Are Natural And Zero-shot Speech And Singing Synthesizers (2023)0.00
• Batch-normalized Joint Training For Dnn-based Distant Speech Recognition (2017)8.82
• Single And Few-step Diffusion For Generative Speech Enhancement (2023)10.21