AV-SAM: Segment Anything Model Meets Audio-visual Localization And Segmentation

Abstract

Segment Anything Model (SAM) has recently shown its powerful effectiveness in visual segmentation tasks. However, there is less exploration concerning how SAM works on audio-visual tasks, such as visual sound localization and segmentation. In this work, we propose a simple yet effective audio-visual localization and segmentation framework based on the Segment Anything Model, namely AV-SAM, that can generate sounding object masks corresponding to the audio. Specifically, our AV-SAM simply leverages pixel-wise audio-visual fusion across audio features and visual features from the pre-trained image encoder in SAM to aggregate cross-modal representations. Then, the aggregated cross-modal features are fed into the prompt encoder and mask decoder to generate the final audio-visual segmentation masks. We conduct extensive experiments on Flickr-SoundNet and AVSBench datasets. The results demonstrate that the proposed AV-SAM can achieve competitive performance on sounding object localization an

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymo2023av

Related papers

• Audio Visual Segmentation Through Text Embeddings (2025)1.81
• SAM: A Mamba-2 State-space Audio-language Model (2025)0.00
• Av-sepformer: Cross-attention Sepformer For Audio-visual Target Speaker Extraction (2023)0.00
• A Study On Joint Modeling And Data Augmentation Of Multi-modalities For Audio-visual Scene Classification (2022)5.24
• Jointly Discovering Visual Objects And Spoken Words From Raw Sensory Input (2018)14.27
• Audio-visual Speech Enhancement And Separation By Utilizing Multi-modal Self-supervised Embeddings (2022)8.60
• Contrastive Conditional Latent Diffusion For Audio-visual Segmentation (2023)10.29
• Audio-visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks (2017)17.39