Audioformer: Audio Transformer Learns Audio Feature Representations From Discrete Acoustic Codes

Abstract

We propose a method named AudioFormer,which learns audio feature representations through the acquisition of discrete acoustic codes and subsequently fine-tunes them for audio classification tasks. Initially,we introduce a novel perspective by considering the audio classification task as a form of natural language understanding (NLU). Leveraging an existing neural audio codec model,we generate discrete acoustic codes and utilize them to train a masked language model (MLM),thereby obtaining audio feature representations. Furthermore,we pioneer the integration of a Multi-Positive sample Contrastive (MPC) learning approach. This method enables the learning of joint representations among multiple discrete acoustic codes within the same audio input. In our experiments,we treat discrete acoustic codes as textual data and train a masked language model using a cloze-like methodology,ultimately deriving high-quality audio representations. Notably,the MPC learning technique effectively captures c

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyli2023audioformer

Related papers

• Audio Language Modeling Using Perceptually-guided Discrete Representations (2022)0.00
• Modeling Strategies For Speech Enhancement In The Latent Space Of A Neural Audio Codec (2025)0.00
• Codec-asr: Training Performant Automatic Speech Recognition Systems With Discrete Speech Representations (2024)6.77
• CTAL: Pre-training Cross-modal Transformer For Audio-and-language Representations (2021)7.50
• Multi-modal Transformers Utterance-level Code-switching Detection (2020)0.00
• Dual Transformer Decoder Based Features Fusion Network For Automated Audio Captioning (2023)4.52
• Neural Speech And Audio Coding: Modern AI Technology Meets Traditional Codecs (2024)7.16
• Transformer-based Sequence Labeling For Audio Classification Based On Mfccs (2023)0.00