Audio-visual Speech Codecs: Rethinking Audio-visual Speech Enhancement By Re-synthesis
2022 · Karren Yang, Dejan Markovic, Steven Krenn, et al.
Abstract
Since facial actions such as lip movements contain significant information about speech content, it is not surprising that audio-visual speech enhancement methods are more accurate than their audio-only counterparts. Yet, state-of-the-art approaches still struggle to generate clean, realistic speech without noise artifacts and unnatural distortions in challenging acoustic environments. In this paper, we propose a novel audio-visual speech enhancement framework for high-fidelity telecommunications in AR/VR. Our approach leverages audio-visual speech cues to generate the codes of a neural speech codec, enabling efficient synthesis of clean, realistic speech from noisy signals. Given the importance of speaker-specific cues in speech, we focus on developing personalized models that work well for individual speakers. We demonstrate the efficacy of our approach on a new audio-visual speech dataset collected in an unconstrained, large vocabulary setting, as well as existing audio-visual datas
Authors
(none)
Tags
Stats
Related papers
- La-voce: Low-snr Audio-visual Speech Enhancement Using Neural Vocoders (2022)0.00
- Revise: Self-supervised Speech Resynthesis With Visual Input For Universal And Generalized Speech Enhancement (2022)0.00
- Av2wav: Diffusion-based Re-synthesis From Continuous Self-supervised Features For Audio-visual Speech Enhancement (2023)0.00
- A Comprehensive Review And Taxonomy Of Audio-visual Synchronization Techniques For Realistic Speech Animation (2024)0.00
- Audiovisual Speaker Conversion: Jointly And Simultaneously Transforming Facial Expression And Acoustic Characteristics (2018)4.52
- Audiovisual Speech Synthesis Using Tacotron2 (2020)8.09
- Voicecraft-dub: Automated Video Dubbing With Neural Codec Language Models (2025)0.00
- Audio-visual Speech Enhancement: Architectural Design And Deployment Strategies (2026)0.00