Separating The "chirp" From The "chat": Self-supervised Visual Grounding Of Sound And Language

Abstract

We present DenseAV, a novel dual encoder grounding architecture that learns high-resolution, semantically meaningful, and audio-visually aligned features solely through watching videos. We show that DenseAV can discover the ``meaning'' of words and the ``location'' of sounds without explicit localization supervision. Furthermore, it automatically discovers and distinguishes between these two types of associations without supervision. We show that DenseAV's localization abilities arise from a new multi-head feature aggregation operator that directly compares dense image and audio representations for contrastive learning. In contrast, many other systems that learn ``global'' audio and video representations cannot localize words and sound. Finally, we contribute two new datasets to improve the evaluation of AV representations through speech and sound prompted semantic segmentation. On these and other datasets we show DenseAV dramatically outperforms the prior art on speech and sound promp

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keyhamilton2024separating

Related papers

• Objects That Sound (2017)0.00
• Avlnet: Learning Audio-visual Language Representations From Instructional Videos (2020)12.87
• Jointly Discovering Visual Objects And Spoken Words From Raw Sensory Input (2018)14.27
• Quality Over Quantity? Llm-based Curation For A Data-efficient Audio-video Foundation Model (2025)0.00
• See, Hear, And Read: Deep Aligned Representations (2017)0.00
• Watch And Learn: Mapping Language And Noisy Real-world Videos With Self-supervision (2020)0.00
• Learning Hierarchical Discrete Linguistic Units From Visually-grounded Speech (2019)0.00
• Sequential Contrastive Audio-visual Learning (2024)5.84