Investigating Modality Bias In Audio Visual Video Parsing

Abstract

We focus on the audio-visual video parsing (AVVP) problem that involves detecting audio and visual event labels with temporal boundaries. The task is especially challenging since it is weakly supervised with only event labels available as a bag of labels for each video. An existing state-of-the-art model for AVVP uses a hybrid attention network (HAN) to generate cross-modal features for both audio and visual modalities, and an attentive pooling module that aggregates predicted audio and visual segment-level event probabilities to yield video-level event probabilities. We provide a detailed analysis of modality bias in the existing HAN architecture, where a modality is completely ignored during prediction. We also propose a variant of feature aggregation in HAN that leads to an absolute gain in F-scores of about 2% and 1.6% for visual and audio-visual events at both segment-level and event-level, in comparison to the existing HAN model.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypasi2022investigating

Related papers

• Cross-modal Learning For Audio-visual Video Parsing (2021)5.84
• Label-anticipated Event Disentanglement For Audio-visual Video Parsing (2024)8.60
• Modality-independent Teachers Meet Weakly-supervised Audio-visual Event Parser (2023)4.77
• Audio Visual Segmentation Through Text Embeddings (2025)1.81
• Audio-visual Event Localization On Portrait Mode Short Videos (2025)0.00
• Efficient Selective Audio Masked Multimodal Bottleneck Transformer For Audio-video Classification (2024)0.00
• Attentive Modality Hopping Mechanism For Speech Emotion Recognition (2019)0.00
• Audio-visual Speech Enhancement And Separation By Utilizing Multi-modal Self-supervised Embeddings (2022)8.60