Avlnet: Learning Audio-visual Language Representations From Instructional Videos

Abstract

Current methods for learning visually grounded language from videos often rely on text annotation, such as human generated captions or machine generated automatic speech recognition (ASR) transcripts. In this work, we introduce the Audio-Video Language Network (AVLnet), a self-supervised network that learns a shared audio-visual embedding space directly from raw video inputs. To circumvent the need for text annotation, we learn audio-visual representations from randomly segmented video clips and their raw audio waveforms. We train AVLnet on HowTo100M, a large corpus of publicly available instructional videos, and evaluate on image retrieval and video retrieval tasks, achieving state-of-the-art performance. We perform analysis of AVLnet's learned representations, showing our model utilizes speech and natural sounds to learn audio-visual concepts. Further, we propose a tri-modal model that jointly processes raw audio, video, and text captions from videos to learn a multi-modal semantic e

Stats

• citations51
• S2 citations—
• github stars0
• HF likes0
• heat score12.87
• arxiv keyrouditchenko2020avlnet

Related papers

• Audio-visual Speech Separation In Noisy Environments With A Lightweight Iterative Model (2023)0.00
• XLAVS-R: Cross-lingual Audio-visual Speech Representation Learning For Noise-robust Speech Perception (2024)7.50
• Audio Visual Segmentation Through Text Embeddings (2025)1.81
• How To Teach Dnns To Pay Attention To The Visual Modality In Speech Recognition (2020)10.97
• Multichannel Av-wav2vec2: A Framework For Learning Multichannel Multi-modal Speech Representation (2024)7.16
• Avformer: Injecting Vision Into Frozen Speech Models For Zero-shot AV-ASR (2023)7.81
• Robust End-to-end Deep Audiovisual Speech Recognition (2016)0.00
• Improved Lite Audio-visual Speech Enhancement (2020)11.39