Muq: Self-supervised Music Representation Learning With Mel Residual Vector Quantization

Abstract

Recent years have witnessed the success of foundation models pre-trained with self-supervised learning (SSL) in various music informatics understanding tasks, including music tagging, instrument classification, key detection, and more. In this paper, we propose a self-supervised music representation learning model for music understanding. Distinguished from previous studies adopting random projection or existing neural codec, the proposed model, named MuQ, is trained to predict tokens generated by Mel Residual Vector Quantization (Mel-RVQ). Our Mel-RVQ utilizes residual linear projection structure for Mel spectrum quantization to enhance the stability and efficiency of target extraction and lead to better performance. Experiments in a large variety of downstream tasks demonstrate that MuQ outperforms previous self-supervised music representation models with only 0.9K hours of open-source pre-training data. Scaling up the data to over 160K hours and adopting iterative training consisten

Stats

• citations25
• S2 citations—
• github stars334
• HF likes0
• heat score15.66
• arxiv keyzhu2025muq

Related papers

• Learning Music Audio Representations Via Weak Language Supervision (2021)10.07
• Musetok: Symbolic Music Tokenization For Generation And Semantic Understanding (2025)0.00
• Supervised And Unsupervised Learning Of Audio Representations For Music Understanding (2022)0.00
• Multi-class-token Transformer For Multitask Self-supervised Music Information Retrieval (2025)0.00
• QS-TTS: Towards Semi-supervised Text-to-speech Synthesis Via Vector-quantized Self-supervised Speech Representation Learning (2023)2.26
• Mumu-llama: Multi-modal Music Understanding And Generation Via Large Language Models (2024)6.34
• Speech Enhancement Using Self-supervised Pre-trained Model And Vector Quantization (2022)6.34
• Supervised Metric Learning For Music Structure Features (2021)0.00