Audio-based Near-duplicate Video Retrieval With Audio Similarity Learning

Abstract

In this work, we address the problem of audio-based near-duplicate video retrieval. We propose the Audio Similarity Learning (AuSiL) approach that effectively captures temporal patterns of audio similarity between video pairs. For the robust similarity calculation between two videos, we first extract representative audio-based video descriptors by leveraging transfer learning based on a Convolutional Neural Network (CNN) trained on a large scale dataset of audio events, and then we calculate the similarity matrix derived from the pairwise similarity of these descriptors. The similarity matrix is subsequently fed to a CNN network that captures the temporal structures existing within its content. We train our network following a triplet generation process and optimizing the triplet loss function. To evaluate the effectiveness of the proposed approach, we have manually annotated two publicly available video datasets based on the audio duplicity between their videos. The proposed approach

Stats

• citations8
• S2 citations—
• github stars0
• HF likes0
• heat score7.16
• arxiv keyavgoustinakis2020audio

Related papers

• Content-based Representations Of Audio Using Siamese Neural Networks (2017)10.61
• CNN Retrieval Based Unsupervised Metric Learning For Near-duplicated Video Retrieval (2021)0.00
• Visil: Fine-grained Spatio-temporal Video Similarity Learning (2019)13.70
• Musical Audio Similarity With Self-supervised Convolutional Neural Networks (2022)0.00
• 3D-CSL: Self-supervised 3D Context Similarity Learning For Near-duplicate Video Retrieval (2022)6.34
• Estimated Audio-caption Correspondences Improve Language-based Audio Retrieval (2024)0.00
• Audio-visual Embedding For Cross-modal Musicvideo Retrieval Through Supervised Deep CCA (2019)11.93
• Deep Triplet Neural Networks With Cluster-cca For Audio-visual Cross-modal Retrieval (2019)12.61