Audio-visual Embedding For Cross-modal Musicvideo Retrieval Through Supervised Deep CCA

Abstract

Deep learning has successfully shown excellent performance in learning joint representations between different data modalities. Unfortunately, little research focuses on cross-modal correlation learning where temporal structures of different data modalities, such as audio and video, should be taken into account. Music video retrieval by given musical audio is a natural way to search and interact with music contents. In this work, we study cross-modal music video retrieval in terms of emotion similarity. Particularly, audio of an arbitrary length is used to retrieve a longer or full-length music video. To this end, we propose a novel audio-visual embedding algorithm by Supervised Deep CanonicalCorrelation Analysis (S-DCCA) that projects audio and video into a shared space to bridge the semantic gap between audio and video. This also preserves the similarity between audio and visual contents from different videos with the same class label and the temporal structure. The contribution of o

Stats

• citations38
• S2 citations—
• github stars0
• HF likes0
• heat score11.93
• arxiv keyzeng2019audio

Related papers

• Variational Autoencoder With CCA For Audio-visual Cross-modal Retrieval (2021)9.92
• Deep Triplet Neural Networks With Cluster-cca For Audio-visual Cross-modal Retrieval (2019)12.61
• Learning Joint Embedding For Cross-modal Retrieval (2019)5.84
• Cross-modal Embeddings For Video And Audio Retrieval (2018)11.08
• Deep Cross-modal Correlation Learning For Audio And Lyrics In Music Retrieval (2017)14.06
• Content-based Video-music Retrieval Using Soft Intra-modal Structure Constraint (2017)3.60
• End-to-end Cross-modality Retrieval With CCA Projections And Pairwise Ranking Loss (2017)14.68
• VMCML: Video And Music Matching Via Cross-modality Lifting (2023)2.26