MAVD: The First Open Large-scale Mandarin Audio-visual Dataset With Depth Information

Abstract

Audio-visual speech recognition (AVSR) gains increasing attention from researchers as an important part of human-computer interaction. However, the existing available Mandarin audio-visual datasets are limited and lack the depth information. To address this issue, this work establishes the MAVD, a new large-scale Mandarin multimodal corpus comprising 12,484 utterances spoken by 64 native Chinese speakers. To ensure the dataset covers diverse real-world scenarios, a pipeline for cleaning and filtering the raw text material has been developed to create a well-balanced reading material. In particular, the latest data acquisition device of Microsoft, Azure Kinect is used to capture depth information in addition to the traditional audio signals and RGB images during data acquisition. We also provide a baseline experiment, which could be used to evaluate the effectiveness of the dataset. The dataset and code will be released at https://github.com/SpringHuo/MAVD.

Stats

• citations2
• S2 citations—
• github stars20
• HF likes0
• heat score6.22
• arxiv keywang2023mavd

Code

• SpringHuo/MAVD—

Related papers

• OLKAVS: An Open Large-scale Korean Audio-visual Speech Dataset (2023)4.52
• Chinese-lips: A Chinese Audio-visual Speech Recognition Dataset With Lip-reading And Presentation Slides (2025)0.00
• Improved Lite Audio-visual Speech Enhancement (2020)11.39
• Robust End-to-end Deep Audiovisual Speech Recognition (2016)0.00
• XLAVS-R: Cross-lingual Audio-visual Speech Representation Learning For Noise-robust Speech Perception (2024)7.50
• Audio Caption: Listen And Tell (2019)10.97
• Auto-acd: A Large-scale Dataset For Audio-language Representation Learning (2023)10.74
• Mscenespeech: A Multi-scene Speech Dataset For Expressive Speech Synthesis (2024)0.00