I See What You Hear: A Vision-inspired Method To Localize Words

Abstract

This paper explores the possibility of using visual object detection techniques for word localization in speech data. Object detection has been thoroughly studied in the contemporary literature for visual data. Noting that an audio can be interpreted as a 1-dimensional image, object localization techniques can be fundamentally useful for word localization. Building upon this idea, we propose a lightweight solution for word detection and localization. We use bounding box regression for word localization, which enables our model to detect the occurrence, offset, and duration of keywords in a given audio stream. We experiment with LibriSpeech and train a model to localize 1000 words. Compared to existing work, our method reduces model size by 94%, and improves the F1 score by 6.5%.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysamragh2022i

Related papers

• Speechyolo: Detection And Localization Of Speech Objects (2019)9.41
• Jointly Discovering Visual Objects And Spoken Words From Raw Sensory Input (2018)14.27
• Keyword Localisation In Untranscribed Speech Using Visually Grounded Speech Models (2022)6.34
• Fine-grained Grounding For Multimodal Speech Recognition (2020)5.84
• Towards Localisation Of Keywords In Speech Using Weak Supervision (2020)0.00
• Learning Word-like Units From Joint Audio-visual Analysis (2017)12.33
• Predicting Detection Filters For Small Footprint Open-vocabulary Keyword Spotting (2019)9.92
• Attention And Localization Based On A Deep Convolutional Recurrent Model For Weakly Supervised Audio Tagging (2017)11.39