Push-pull: Characterizing The Adversarial Robustness For Audio-visual Active Speaker Detection

Abstract

Audio-visual active speaker detection (AVASD) is well-developed, and now is an indispensable front-end for several multi-modal applications. However, to the best of our knowledge, the adversarial robustness of AVASD models hasn't been investigated, not to mention the effective defense against such attacks. In this paper, we are the first to reveal the vulnerability of AVASD models under audio-only, visual-only, and audio-visual adversarial attacks through extensive experiments. What's more, we also propose a novel audio-visual interaction loss (AVIL) for making attackers difficult to find feasible adversarial examples under an allocated attack budget. The loss aims at pushing the inter-class embeddings to be dispersed, namely non-speech and speech clusters, sufficiently disentangled, and pulling the intra-class embeddings as close as possible to keep them compact. Experimental results show the AVIL outperforms the adversarial training by 33.14 mAP (%) under multi-modal attacks.

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keychen2022push

Related papers

• Diffusion-based Adversarial Purification For Speaker Verification (2023)6.34
• Enhancing Real-world Active Speaker Detection With Multi-modal Extraction Pre-training (2024)5.24
• Adversarial Attack And Defense Strategies For Deep Speaker Recognition Systems (2020)13.39
• Adversarial Sample Detection For Speaker Verification By Neural Vocoders (2021)0.00
• A Joint Noise Disentanglement And Adversarial Training Framework For Robust Speaker Verification (2024)6.34
• Toward Improving Synthetic Audio Spoofing Detection Robustness Via Meta-learning And Disentangled Training With Adversarial Examples (2024)6.77
• Inaudible Adversarial Perturbations For Targeted Attack In Speaker Recognition (2020)12.33
• Active Speaker Detection As A Multi-objective Optimization With Uncertainty-based Multimodal Fusion (2021)7.50