Emotioncaps: Enhancing Audio Captioning Through Emotion-augmented Data Generation

Abstract

Recent progress in audio-language modeling, such as automated audio captioning, has benefited from training on synthetic data generated with the aid of large-language models. However, such approaches for environmental sound captioning have primarily focused on audio event tags and have not explored leveraging emotional information that may be present in recordings. In this work, we explore the benefit of generating emotion-augmented synthetic audio caption data by instructing ChatGPT with additional acoustic information in the form of estimated soundscape emotion. To do so, we introduce EmotionCaps, an audio captioning dataset comprised of approximately 120,000 audio clips with paired synthetic descriptions enriched with soundscape emotion recognition (SER) information. We hypothesize that this additional information will result in higher-quality captions that match the emotional tone of the audio recording, which will, in turn, improve the performance of captioning models trained with

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymanivannan2024emotioncaps

Related papers

• Sound-vecaps: Improving Audio Generation With Visual Enhanced Captions (2024)7.16
• Audiosetcaps: An Enriched Audio-caption Dataset Using Automated Generation Pipeline With Large Audio And Language Models (2024)13.44
• Generative Emotional AI For Speech Emotion Recognition: The Case For Synthetic Emotional Speech Augmentation (2023)11.19
• Muscaps: Generating Captions For Music Audio (2021)9.59
• RECAP: Retrieval-augmented Audio Captioning (2023)9.41
• Taming Data And Transformers For Audio Generation (2024)0.00
• Automatically Augmenting An Emotion Dataset Improves Classification Using Audio (2018)0.00
• Wavcaps: A Chatgpt-assisted Weakly-labelled Audio Captioning Dataset For Audio-language Multimodal Research (2023)20.69