Enhance Audio Generation Controllability Through Representation Similarity Regularization

Abstract

This paper presents an innovative approach to enhance control over audio generation by emphasizing the alignment between audio and text representations during model training. In the context of language model-based audio generation, the model leverages input from both textual and audio token representations to predict subsequent audio tokens. However, the current configuration lacks explicit regularization to ensure the alignment between the chosen text representation and the language model's predictions. Our proposal involves the incorporation of audio and text representation regularization, particularly during the classifier-free guidance (CFG) phase, where the text condition is excluded from cross attention during language model training. The aim of this proposed representation regularization is to minimize discrepancies in audio and text similarity compared to other samples within the same training batch. Experimental results on both music and audio generation tasks demonstrate that

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyshi2023enhance

Related papers

• Cross-lingual Spoken Language Understanding With Regularized Representation Alignment (2020)6.77
• Audiogen: Textually Guided Audio Generation (2022)0.00
• Audiomog: Guiding Audio Generation With Mixture-of-guidance (2025)0.00
• Diverse And Aligned Audio-to-video Generation Via Text-to-video Model Adaptation (2023)11.19
• Audio Language Modeling Using Perceptually-guided Discrete Representations (2022)0.00
• Multimodal Representation Loss Between Timed Text And Audio For Regularized Speech Separation (2024)0.00
• Audiorag+: Feedback-driven Retrieval-augmented Audio Generation With Large Audio Language Models (2025)0.00
• Audiolm: A Language Modeling Approach To Audio Generation (2022)18.91