Large-scale Contrastive Language-audio Pretraining With Feature Fusion And Keyword-to-caption Augmentation

Abstract

Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text

Stats

• citations410
• S2 citations—
• github stars0
• HF likes0
• heat score19.60
• arxiv keywu2022large

Related papers

• Audiosetcaps: An Enriched Audio-caption Dataset Using Automated Generation Pipeline With Large Audio And Language Models (2024)13.44
• Collap: Contrastive Long-form Language-audio Pretraining With Musical Temporal Structure Augmentation (2024)3.58
• Audio Captioning Using Pre-trained Large-scale Language Model Guided By Audio-based Similar Caption Retrieval (2020)0.00
• Learning Audio-video Modalities From Image Captions (2022)12.54
• Interactive Audio-text Representation For Automated Audio Captioning With Contrastive Learning (2022)0.00
• BLAT: Bootstrapping Language-audio Pre-training Based On Audioset Tag-guided Synthetic Data (2023)8.35
• Auto-acd: A Large-scale Dataset For Audio-language Representation Learning (2023)10.74
• Advancing Natural-language Based Audio Retrieval With Passt And Large Audio-caption Data Sets (2023)0.00