Learning Word-like Units From Joint Audio-visual Analysis

Abstract

Given a collection of images and spoken audio captions, we present a method for discovering word-like acoustic units in the continuous speech signal and grounding them to semantically relevant image regions. For example, our model is able to detect spoken instances of the word 'lighthouse' within an utterance and associate them with image regions containing lighthouses. We do not use any form of conventional automatic speech recognition, nor do we use any text transcriptions or conventional linguistic annotations. Our model effectively implements a form of spoken language acquisition, in which the computer learns not only to recognize word categories by sound, but also to enrich the words it learns with semantics by grounding them in images.

Stats

• citations43
• S2 citations—
• github stars0
• HF likes0
• heat score12.33
• arxiv keyharwath2017learning

Related papers

• Jointly Discovering Visual Objects And Spoken Words From Raw Sensory Input (2018)14.27
• Language Learning Using Speech To Image Retrieval (2019)9.41
• Learning Hierarchical Discrete Linguistic Units From Visually-grounded Speech (2019)0.00
• Text-free Image-to-speech Synthesis Using Learned Segmental Units (2020)10.85
• Towards Visually Grounded Sub-word Speech Unit Discovery (2019)9.03
• Fine-grained Grounding For Multimodal Speech Recognition (2020)5.84
• Word Recognition, Competition, And Activation In A Model Of Visually Grounded Speech (2019)0.00
• Towards Language Modelling In The Speech Domain Using Sub-word Linguistic Units (2021)0.00