Object Referring In Visual Scene With Spoken Language

Abstract

Object referring has important applications, especially for human-machine interaction. While having received great attention, the task is mainly attacked with written language (text) as input rather than spoken language (speech), which is more natural. This paper investigates Object Referring with Spoken Language (ORSpoken) by presenting two datasets and one novel approach. Objects are annotated with their locations in images, text descriptions and speech descriptions. This makes the datasets ideal for multi-modality learning. The approach is developed by carefully taking down ORSpoken problem into three sub-problems and introducing task-specific vision-language interactions at the corresponding levels. Experiments show that our method outperforms competing methods consistently and significantly. The approach is also evaluated in the presence of audio noise, showing the efficacy of the proposed vision-language interaction methods in counteracting background noise.

Stats

• citations8
• S2 citations—
• github stars0
• HF likes0
• heat score7.16
• arxiv keyvasudevan2017object

Related papers

• Jointly Discovering Visual Objects And Spoken Words From Raw Sensory Input (2018)14.27
• Semantic Speech Retrieval With A Visually Grounded Model Of Untranscribed Speech (2017)10.61
• Expression Prompt Collaboration Transformer For Universal Referring Video Object Segmentation (2023)4.52
• Fine-grained Grounding For Multimodal Speech Recognition (2020)5.84
• Spatial Concept Acquisition For A Mobile Robot That Integrates Self-localization And Unsupervised Word Discovery From Spoken Sentences (2016)9.92
• Language Learning Using Speech To Image Retrieval (2019)9.41
• Speechyolo: Detection And Localization Of Speech Objects (2019)9.41
• Open Vocabulary Keyword Spotting Through Transfer Learning From Speech Synthesis (2024)0.00