Automated Audio Captioning With Recurrent Neural Networks

Abstract

We present the first approach to automated audio captioning. We employ an encoder-decoder scheme with an alignment model in between. The input to the encoder is a sequence of log mel-band energies calculated from an audio file, while the output is a sequence of words, i.e. a caption. The encoder is a multi-layered, bi-directional gated recurrent unit (GRU) and the decoder a multi-layered GRU with a classification layer connected to the last GRU of the decoder. The classification layer and the alignment model are fully connected layers with shared weights between timesteps. The proposed method is evaluated using data drawn from a commercial sound effects library, ProSound Effects. The resulting captions were rated through metrics utilized in machine translation and image captioning fields. Results from metrics show that the proposed method can predict words appearing in the original caption, but not always correctly ordered.

Stats

• citations72
• S2 citations—
• github stars0
• HF likes0
• heat score13.97
• arxiv keydrossos2017automated

Related papers

• An Encoder-decoder Based Audio Captioning System With Transfer And Reinforcement Learning (2021)0.00
• Listen Carefully And Tell: An Audio Captioning System Based On Residual Learning And Gammatone Audio Representation (2020)0.00
• Audio Captioning Using Gated Recurrent Units (2020)0.00
• Automated Audio Captioning: An Overview Of Recent Progress And New Challenges (2022)12.10
• Temporal Sub-sampling Of Audio Feature Sequences For Automated Audio Captioning (2020)0.00
• Improving The Performance Of Automated Audio Captioning Via Integrating The Acoustic And Semantic Information (2021)2.00
• RECAP: Retrieval-augmented Audio Captioning (2023)9.41
• Automatic Audio Captioning Using Attention Weighted Event Based Embeddings (2022)0.00