AV2AV: Direct Audio-visual Speech To Audio-visual Speech Translation With Unified Audio-visual Speech Representation

Abstract

This paper proposes a novel direct Audio-Visual Speech to Audio-Visual Speech Translation (AV2AV) framework, where the input and output of the system are multimodal (i.e., audio and visual speech). With the proposed AV2AV, two key advantages can be brought: 1) We can perform real-like conversations with individuals worldwide in a virtual meeting by utilizing our own primary languages. In contrast to Speech-to-Speech Translation (A2A), which solely translates between audio modalities, the proposed AV2AV directly translates between audio-visual speech. This capability enhances the dialogue experience by presenting synchronized lip movements along with the translated speech. 2) We can improve the robustness of the spoken language translation system. By employing the complementary information of audio-visual speech, the system can effectively translate spoken language even in the presence of acoustic noise, showcasing robust performance. To mitigate the problem of the absence of a parallel

Stats

• citations7
• S2 citations—
• github stars0
• HF likes0
• heat score6.77
• arxiv keychoi2023av2av

Related papers

• XLAVS-R: Cross-lingual Audio-visual Speech Representation Learning For Noise-robust Speech Perception (2024)7.50
• Visualtts: TTS With Accurate Lip-speech Synchronization For Automatic Voice Over (2021)9.41
• Improving Lip-synchrony In Direct Audio-visual Speech-to-speech Translation (2024)0.00
• Av-link: Temporally-aligned Diffusion Features For Cross-modal Audio-video Generation (2024)0.00
• Audio Visual Segmentation Through Text Embeddings (2025)1.81
• Improved Lite Audio-visual Speech Enhancement (2020)11.39
• Audio-visual Speech Codecs: Rethinking Audio-visual Speech Enhancement By Re-synthesis (2022)15.58
• Multi-layer Feature Fusion Convolution Network For Audio-visual Speech Enhancement (2021)0.00