Word Recognition, Competition, And Activation In A Model Of Visually Grounded Speech

Abstract

In this paper, we study how word-like units are represented and activated in a recurrent neural model of visually grounded speech. The model used in our experiments is trained to project an image and its spoken description in a common representation space. We show that a recurrent model trained on spoken sentences implicitly segments its input into word-like units and reliably maps them to their correct visual referents. We introduce a methodology originating from linguistics to analyse the representation learned by neural networks -- the gating paradigm -- and show that the correct representation of a word is only activated if the network has access to first phoneme of the target word, suggesting that the network does not rely on a global acoustic pattern. Furthermore, we find out that not all speech frames (MFCC vectors in our case) play an equal role in the final encoded representation of a given word, but that some frames have a crucial effect on it. Finally, we suggest that word r

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhavard2019word

Related papers

• Representations Of Language In A Model Of Visually Grounded Speech Signal (2017)12.02
• Catplayinginthesnow: Impact Of Prior Segmentation On A Model Of Visually Grounded Speech (2020)4.52
• Towards Visually Grounded Sub-word Speech Unit Discovery (2019)9.03
• Word Discovery In Visually Grounded, Self-supervised Speech Models (2022)14.08
• Language Learning Using Speech To Image Retrieval (2019)9.41
• Learning Hierarchical Discrete Linguistic Units From Visually-grounded Speech (2019)0.00
• Fine-grained Grounding For Multimodal Speech Recognition (2020)5.84
• Learning Word-like Units From Joint Audio-visual Analysis (2017)12.33