Investigating Self-supervised Representations For Audio-visual Deepfake Detection

Abstract

Self-supervised representations excel at many vision and speech tasks, but their potential for audio-visual deepfake detection remains underexplored. Unlike prior work that uses these features in isolation or buried within complex architectures, we systematically evaluate them across modalities (audio, video, multimodal) and domains (lip movements, generic visual content). We assess three key dimensions: detection effectiveness, interpretability of encoded information, and cross-modal complementarity. We find that most self-supervised features capture deepfake-relevant information, and that this information is complementary. Moreover, models primarily attend to semantically meaningful regions rather than spurious artifacts (such as the leading silence). Among the investigated features, audio-informed representations generalize best and achieve state-of-the-art results. However, generalization to realistic in-the-wild data remains challenging. Our analysis indicates this gap stems from

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyboldisor2025investigating

Related papers

• Towards Generalisable And Calibrated Synthetic Speech Detection With Self-supervised Representations (2023)0.00
• Learning Speech Representations From Raw Audio By Joint Audiovisual Self-supervision (2020)0.00
• ERF-BA-TFD+: A Multimodal Model For Audio-visual Deepfake Detection (2025)2.26
• Temporal Variability And Multi-viewed Self-supervised Representations To Tackle The Asvspoof5 Deepfake Challenge (2024)0.00
• Self-attention And Hybrid Features For Replay And Deep-fake Audio Detection (2024)0.00
• Combining Automatic Speaker Verification And Prosody Analysis For Synthetic Speech Detection (2022)10.48
• Adaptive Re-calibration Of Channel-wise Features For Adversarial Audio Classification (2022)0.00
• Avtenet: A Human-cognition-inspired Audio-visual Transformer-based Ensemble Network For Video Deepfake Detection (2023)7.50