Fine-grained Instance-level Sketch-based Video Retrieval

Abstract

Existing sketch-analysis work studies sketches depicting static objects or scenes. In this work, we propose a novel cross-modal retrieval problem of fine-grained instance-level sketch-based video retrieval (FG-SBVR), where a sketch sequence is used as a query to retrieve a specific target video instance. Compared with sketch-based still image retrieval, and coarse-grained category-level video retrieval, this is more challenging as both visual appearance and motion need to be simultaneously matched at a fine-grained level. We contribute the first FG-SBVR dataset with rich annotations. We then introduce a novel multi-stream multi-modality deep network to perform FG-SBVR under both strong and weakly supervised settings. The key component of the network is a relation module, designed to prevent model over-fitting given scarce training data. We show that this model significantly outperforms a number of existing state-of-the-art models designed for video analysis.

Stats

• citations34
• S2 citations—
• github stars0
• HF likes0
• heat score11.58
• arxiv keyxu2020fine

Related papers

• Freeview Sketching: View-aware Fine-grained Sketch-based Image Retrieval (2024)6.34
• Sketch Less For More: On-the-fly Fine-grained Sketch Based Image Retrieval (2020)15.28
• Deep Reinforced Attention Regression For Partial Sketch Based Image Retrieval (2021)5.24
• Adaptive Fine-grained Sketch-based Image Retrieval (2022)9.76
• Cross-modal Subspace Learning For Fine-grained Sketch-based Image Retrieval (2017)13.34
• Cross-modal Hierarchical Modelling For Fine-grained Sketch Based Image Retrieval (2020)6.77
• More Photos Are All You Need: Semi-supervised Learning For Fine-grained Sketch Based Image Retrieval (2021)13.23
• Arnet: Self-supervised FG-SBIR With Unified Sample Feature Alignment And Multi-scale Token Recycling (2024)5.84