Deep Cross-modal Correlation Learning For Audio And Lyrics In Music Retrieval

Abstract

Little research focuses on cross-modal correlation learning where temporal structures of different data modalities such as audio and lyrics are taken into account. Stemming from the characteristic of temporal structures of music in nature, we are motivated to learn the deep sequential correlation between audio and lyrics. In this work, we propose a deep cross-modal correlation learning architecture involving two-branch deep neural networks for audio modality and text modality (lyrics). Different modality data are converted to the same canonical space where inter modal canonical correlation analysis is utilized as an objective function to calculate the similarity of temporal structures. This is the first study on understanding the correlation between language and music audio through deep architectures for learning the paired temporal correlation of audio and lyrics. Pre-trained Doc2vec model followed by fully-connected layers (fully-connected deep neural network) is used to represent ly

Stats

• citations74
• S2 citations—
• github stars0
• HF likes0
• heat score14.06
• arxiv keyyu2017deep

Related papers

• Audio-visual Embedding For Cross-modal Musicvideo Retrieval Through Supervised Deep CCA (2019)11.93
• Learning Joint Embedding For Cross-modal Retrieval (2019)5.84
• Passage Summarization With Recurrent Models For Audio-sheet Music Retrieval (2023)0.00
• Content-based Video-music Retrieval Using Soft Intra-modal Structure Constraint (2017)3.60
• Towards Robust And Truly Large-scale Audio-sheet Music Retrieval (2023)4.52
• Deep Triplet Neural Networks With Cluster-cca For Audio-visual Cross-modal Retrieval (2019)12.61
• Contrastive Audio-language Learning For Music (2022)0.00
• Towards End-to-end Audio-sheet-music Retrieval (2016)0.00