Content-based Video-music Retrieval Using Soft Intra-modal Structure Constraint

Abstract

Up to now, only limited research has been conducted on cross-modal retrieval of suitable music for a specified video or vice versa. Moreover, much of the existing research relies on metadata such as keywords, tags, or associated description that must be individually produced and attached posterior. This paper introduces a new content-based, cross-modal retrieval method for video and music that is implemented through deep neural networks. We train the network via inter-modal ranking loss such that videos and music with similar semantics end up close together in the embedding space. However, if only the inter-modal ranking constraint is used for embedding, modality-specific characteristics can be lost. To address this problem, we propose a novel soft intra-modal structure loss that leverages the relative distance relationship between intra-modal samples before embedding. We also introduce reasonable quantitative and qualitative experimental protocols to solve the lack of standard protoco

Stats

• citations0
• S2 citations—
• github stars62
• HF likes0
• heat score3.60
• arxiv keyhong2017content

Related papers

• Audio-visual Embedding For Cross-modal Musicvideo Retrieval Through Supervised Deep CCA (2019)11.93
• VMCML: Video And Music Matching Via Cross-modality Lifting (2023)2.26
• Contrastive Learning For Cross-modal Artist Retrieval (2023)0.00
• Deep Music Retrieval For Fine-grained Videos By Exploiting Cross-modal-encoded Voice-overs (2021)6.34
• Deep Cross-modal Correlation Learning For Audio And Lyrics In Music Retrieval (2017)14.06
• Cross-modal Embeddings For Video And Audio Retrieval (2018)11.08
• A Multimodal Deep Learning Framework For Scalable Content Based Visual Media Retrieval (2021)0.00
• Video And Audio Are Images: A Cross-modal Mixer For Original Data On Video-audio Retrieval (2023)7.16