Unsupervised Audio-caption Aligning Learns Correspondences Between Individual Sound Events And Textual Phrases

Abstract

We investigate unsupervised learning of correspondences between sound events and textual phrases through aligning audio clips with textual captions describing the content of a whole audio clip. We align originally unaligned and unannotated audio clips and their captions by scoring the similarities between audio frames and words, as encoded by modality-specific encoders and using a ranking-loss criterion to optimize the model. After training, we obtain clip-caption similarity by averaging frame-word similarities and estimate event-phrase correspondences by calculating frame-phrase similarities. We evaluate the method with two cross-modal tasks: audio-caption retrieval, and phrase-based sound event detection (SED). Experimental results show that the proposed method can globally associate audio clips with captions as well as locally learn correspondences between individual sound events and textual phrases in an unsupervised manner.

Stats

• citations11
• S2 citations—
• github stars0
• HF likes0
• heat score8.09
• arxiv keyxie2021unsupervised

Related papers

• Estimated Audio-caption Correspondences Improve Language-based Audio Retrieval (2024)0.00
• Text-based Audio Retrieval By Learning From Similarities Between Audio Captions (2024)2.26
• Automated Audio Captioning With Recurrent Neural Networks (2017)13.97
• Investigations In Audio Captioning: Addressing Vocabulary Imbalance And Evaluating Suitability Of Language-centric Performance Metrics (2022)0.00
• Learning Audio-video Modalities From Image Captions (2022)12.54
• Audio Caption: Listen And Tell (2019)10.97
• Audio Difference Captioning Utilizing Similarity-discrepancy Disentanglement (2023)2.26
• Interactive Audio-text Representation For Automated Audio Captioning With Contrastive Learning (2022)0.00