Fuse After Align: Improving Face-voice Association Learning Via Multimodal Encoder

Abstract

Today, there have been many achievements in learning the association between voice and face. However, most previous work models rely on cosine similarity or L2 distance to evaluate the likeness of voices and faces following contrastive learning, subsequently applied to retrieval and matching tasks. This method only considers the embeddings as high-dimensional vectors, utilizing a minimal scope of available information. This paper introduces a novel framework within an unsupervised setting for learning voice-face associations. By employing a multimodal encoder after contrastive learning and addressing the problem through binary classification, we can learn the implicit information within the embeddings in a more effective and varied manner. Furthermore, by introducing an effective pair selection method, we enhance the learning outcomes of both contrastive learning and the matching task. Empirical evidence demonstrates that our framework achieves state-of-the-art results in voice-face ma

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypeng2024fuse

Related papers

• Seeking The Shape Of Sound: An Adaptive Framework For Learning Voice-face Association (2021)11.39
• Voice-face Cross-modal Matching And Retrieval: A Benchmark (2019)0.00
• The More, The Merrier: Contrastive Fusion For Higher-order Multimodal Alignment (2025)0.00
• Perfect Match: Improved Cross-modal Embeddings For Audio-visual Synchronisation (2018)14.19
• Learnable Pins: Cross-modal Embeddings For Person Identity (2018)15.22
• Come-vl: Scaling Complementary Multi-encoder Vision-language Learning (2026)0.00
• Clover: Towards A Unified Video-language Alignment And Fusion Model (2022)14.30
• Hyperdimensional Cross-modal Alignment Of Frozen Language And Image Models For Efficient Image Captioning (2026)0.00