← all papers · overview

Deep Music Retrieval For Fine-grained Videos By Exploiting Cross-modal-encoded Voice-overs

2021 · Tingtian Li, Zixun Sun, Haoruo Zhang, et al.

Abstract

Recently, the witness of the rapidly growing popularity of short videos on different Internet platforms has intensified the need for a background music (BGM) retrieval system. However, existing video-music retrieval methods only based on the visual modality cannot show promising performance regarding videos with fine-grained virtual contents. In this paper, we also investigate the widely added voice-overs in short videos and propose a novel framework to retrieve BGM for fine-grained short videos. In our framework, we use the self-attention (SA) and the cross-modal attention (CMA) modules to explore the intra- and the inter-relationships of different modalities respectively. For balancing the modalities, we dynamically assign different weights to the modal features via a fusion gate. For paring the query and the BGM embeddings, we introduce a triplet pseudo-label loss to constrain the semantics of the modal embeddings. As there are no existing virtual-content video-BGM retrieval dataset

Authors

(none)

Tags

  • Cross-Modal Hashing
  • Image Retrieval

Stats

  • citations6
  • S2 citations—
  • github stars0
  • HF likes0
  • heat score6.34
  • arxiv keyli2021deep

Related papers

Deep Music Retrieval For Fine-grained Videos By Exploiting Cross-modal-encoded Voice-overs — learning-to-hash