Exploring Train And Test-time Augmentations For Audio-language Learning

Abstract

In this paper, we aim to unveil the impact of data augmentation in audio-language multi-modal learning, which has not been explored despite its importance. We explore various augmentation methods at not only train-time but also test-time and find out that proper data augmentation can lead to substantial improvements. Specifically, applying our proposed audio-language paired augmentation PairMix, which is the first multi-modal audio-language augmentation method, outperforms the baselines for both automated audio captioning and audio-text retrieval tasks. To fully take advantage of data augmentation, we also present multi-level test-time augmentation (Multi-TTA) for the test-time. We successfully incorporate the two proposed methods and uni-modal augmentations and achieve 47.5 SPIDEr on audio captioning, which is an 18.2% relative increase over the baseline. In audio-text retrieval, the proposed methods also show an improvement in performance as well.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykim2022exploring

Related papers

• Audiosetmix: Enhancing Audio-language Datasets With Llm-assisted Augmentations (2024)0.00
• Improving Natural-language-based Audio Retrieval With Transfer Learning And Audio & Text Augmentations (2022)0.00
• Sample Mixed-based Data Augmentation For Domestic Audio Tagging (2018)0.00
• A Study On Joint Modeling And Data Augmentation Of Multi-modalities For Audio-visual Scene Classification (2022)5.24
• Segaugment: Maximizing The Utility Of Speech Translation Data With Segmentation-based Augmentations (2022)0.00
• Improving Multimodal Speech Recognition By Data Augmentation And Speech Representations (2022)9.03
• Sample, Translate, Recombine: Leveraging Audio Alignments For Data Augmentation In End-to-end Speech Translation (2022)7.81
• Mixspeech: Data Augmentation For Low-resource Automatic Speech Recognition (2021)13.60