Dual Transformer Decoder Based Features Fusion Network For Automated Audio Captioning

Abstract

Automated audio captioning (AAC) which generates textual descriptions of audio content. Existing AAC models achieve good results but only use the high-dimensional representation of the encoder. There is always insufficient information learning of high-dimensional methods owing to high-dimensional representations having a large amount of information. In this paper, a new encoder-decoder model called the Low- and High-Dimensional Feature Fusion (LHDFF) is proposed. LHDFF uses a new PANNs encoder called Residual PANNs (RPANNs) to fuse low- and high-dimensional features. Low-dimensional features contain limited information about specific audio scenes. The fusion of low- and high-dimensional features can improve model performance by repeatedly emphasizing specific audio scene information. To fully exploit the fused features, LHDFF uses a dual transformer decoder structure to generate captions in parallel. Experimental results show that LHDFF outperforms existing audio captioning models.

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keysun2023dual

Related papers

• Automated Audio Captioning Via Fusion Of Low- And High- Dimensional Features (2022)0.00
• Wavetransformer: A Novel Architecture For Audio Captioning Based On Learning Temporal And Time-frequency Information (2020)0.00
• Enhancing Automated Audio Captioning Via Large Language Models With Optimized Audio Encoding (2024)5.24
• Improving Audio Captioning Models With Fine-grained Audio Features, Text Embedding Supervision, And LLM Mix-up Augmentation (2023)8.82
• Improving The Performance Of Automated Audio Captioning Via Integrating The Acoustic And Semantic Information (2021)2.00
• Parameter Efficient Audio Captioning With Faithful Guidance Using Audio-text Shared Latent Representation (2023)3.58
• An Encoder-decoder Based Audio Captioning System With Transfer And Reinforcement Learning (2021)0.00
• Conette: An Efficient Audio Captioning System Leveraging Multiple Datasets With Task Embedding (2023)11.11