Singmos-pro: An Comprehensive Benchmark For Singing Quality Assessment

Abstract

Singing voice generation progresses rapidly, yet evaluating singing quality remains a critical challenge. Human subjective assessment, typically in the form of listening tests, is costly and time consuming, while existing objective metrics capture only limited perceptual aspects. In this work, we introduce SingMOS-Pro, a dataset for automatic singing quality assessment. Building on our preview version SingMOS, which provides only overall ratings, SingMOS-Pro extends the annotations of the additional data to include lyrics, melody, and overall quality, offering broader coverage and greater diversity. The dataset contains 7,981 singing clips generated by 41 models across 12 datasets, spanning from early systems to recent state-of-the-art approaches. Each clip is rated by at least five experienced annotators to ensure reliability and consistency. Furthermore, we investigate strategies for effectively utilizing MOS data annotated under heterogeneous standards and benchmark several widely u

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytang2025singmos

Related papers

• Singmos: An Extensive Open-source Singing Voice Dataset For MOS Prediction (2024)0.00
• Pitch-and-spectrum-aware Singing Quality Assessment With Bias Correction And Model Fusion (2024)3.58
• S2cap: A Benchmark And A Baseline For Singing Style Captioning (2024)0.00
• Opencpop: A High-quality Open Source Chinese Popular Song Corpus For Singing Voice Synthesis (2022)13.34
• Xiaoicesing: A High-quality And Integrated Singing Voice Synthesis System (2020)12.54
• Multi-singer: Fast Multi-singer Singing Voice Vocoder With A Large-scale Corpus (2021)13.28
• Singomd: Singing Oriented Multi-resolution Discrete Representation Construction From Speech Models (2024)5.24
• Adversarially Trained Multi-singer Sequence-to-sequence Singing Synthesizer (2020)7.81