Sign Language Video Retrieval With Free-form Textual Queries

Abstract

Systems that can efficiently search collections of sign language videos have been highlighted as a useful application of sign language technology. However, the problem of searching videos beyond individual keywords has received limited attention in the literature. To address this gap, in this work we introduce the task of sign language retrieval with free-form textual queries: given a written query (e.g., a sentence) and a large collection of sign language videos, the objective is to find the signing video in the collection that best matches the written query. We propose to tackle this task by learning cross-modal embeddings on the recently introduced large-scale How2Sign dataset of American Sign Language (ASL). We identify that a key bottleneck in the performance of the system is the quality of the sign video embedding which suffers from a scarcity of labeled training data. We, therefore, propose SPOT-ALIGN, a framework for interleaving iterative rounds of sign spotting and feature al

Stats

• citations23
• S2 citations—
• github stars0
• HF likes0
• heat score10.35
• arxiv keyduarte2022sign

Related papers

• Cico: Domain-aware Sign Language Retrieval Via Cross-lingual Contrastive Learning (2023)16.35
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts (2019)0.00
• A Tale Of Two Languages: Large-vocabulary Continuous Sign Language Recognition From Spoken Language Supervision (2024)0.00
• SEA: Sentence Encoder Assembly For Video Retrieval By Textual Queries (2020)12.47
• The VISIONE Video Search System: Exploiting Off-the-shelf Text Search Engines For Large-scale Video Retrieval (2020)10.74
• Tree-augmented Cross-modal Encoding For Complex-query Video Retrieval (2020)15.57
• Dual Encoding For Video Retrieval By Text (2020)16.05
• Learning Joint Representations Of Videos And Sentences With Web Image Search (2016)12.93