Contrastive Learning-based Chaining-cluster For Multilingual Voice-face Association

Abstract

The innate correlation between a person's face and voice has recently emerged as a compelling area of study, especially within the context of multilingual environments. This paper introduces our novel solution to the Face-Voice Association in Multilingual Environments (FAME) 2024 challenge, focusing on a contrastive learning-based chaining-cluster method to enhance face-voice association. This task involves the challenges of building biometric relations between auditory and visual modality cues and modelling the prosody interdependence between different languages while addressing both intrinsic and extrinsic variability present in the data. To handle these non-trivial challenges, our method employs supervised cross-contrastive (SCC) learning to establish robust associations between voices and faces in multi-language scenarios. Following this, we have specifically designed a chaining-cluster-based post-processing step to mitigate the impact of outliers often found in unconstrained in th

Stats

• citations2
• S2 citations—
• github stars3
• HF likes0
• heat score4.78
• arxiv keychen2024contrastive

Related papers

• Cross-modal Speaker Verification And Recognition: A Multilingual Perspective (2020)0.00
• Face-voice Association In Multilingual Environments (FAME) Challenge 2024 Evaluation Plan (2024)0.00
• PAEFF: Precise Alignment And Enhanced Gated Feature Fusion For Face-voice Association (2025)6.34
• Multi-stage Face-voice Association Learning With Keynote Speaker Diarization (2024)7.97
• Seeing Your Speech Style: A Novel Zero-shot Identity-disentanglement Face-based Voice Conversion (2024)4.52
• Unsupervised Voice-face Representation Learning By Cross-modal Prototype Contrast (2022)10.35
• Listening While Speaking And Visualizing: Improving ASR Through Multimodal Chain (2019)4.52
• Multi-modal Multi-correlation Learning For Audio-visual Speech Separation (2022)5.84