M\(^{2}\)ugen: Multi-modal Music Understanding And Generation With The Power Of Large Language Models

Abstract

The current landscape of research leveraging large language models (LLMs) is experiencing a surge. Many works harness the powerful reasoning capabilities of these models to comprehend various modalities, such as text, speech, images, videos, etc. They also utilize LLMs to understand human intention and generate desired outputs like images, videos, and music. However, research that combines both understanding and generation using LLMs is still limited and in its nascent stage. To address this gap, we introduce a Multi-modal Music Understanding and Generation (M\(^\{2\}\)UGen) framework that integrates LLM's abilities to comprehend and generate music for different modalities. The M\(^\{2\}\)UGen framework is purpose-built to unlock creative potential from diverse sources of inspiration, encompassing music, image, and video through the use of pretrained MERT, ViT, and ViViT models, respectively. To enable music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging multi-moda

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2023m

Related papers

• Mumu-llama: Multi-modal Music Understanding And Generation Via Large Language Models (2024)6.34
• Musilingo: Bridging Music And Text With Pre-trained Language Models For Music Captioning And Query Response (2023)9.03
• Llms Meet Multimodal Generation And Editing: A Survey (2024)5.48
• A Review Of Multi-modal Large Language And Vision Models (2024)0.00
• Uniaudio: An Audio Foundation Model Toward Universal Audio Generation (2023)5.56
• C3LLM: Conditional Multimodal Content Generation Using Large Language Models (2024)0.00
• Gamma: Towards Joint Global-temporal Music Understanding In Large Multimodal Models (2026)0.00
• Musicldm: Enhancing Novelty In Text-to-music Generation Using Beat-synchronous Mixup Strategies (2023)13.55