Catplayinginthesnow: Impact Of Prior Segmentation On A Model Of Visually Grounded Speech

Abstract

The language acquisition literature shows that children do not build their lexicon by segmenting the spoken input into phonemes and then building up words from them, but rather adopt a top-down approach and start by segmenting word-like units and then break them down into smaller units. This suggests that the ideal way of learning a language is by starting from full semantic units. In this paper, we investigate if this is also the case for a neural model of Visually Grounded Speech trained on a speech-image retrieval task. We evaluated how well such a network is able to learn a reliable speech-to-image mapping when provided with phone, syllable, or word boundary information. We present a simple way to introduce such information into an RNN-based model and investigate which type of boundary is the most efficient. We also explore at which level of the network's architecture such information should be introduced so as to maximise its performances. Finally, we show that using multiple boun

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keyhavard2020catplayinginthesnow

Related papers

• Word Recognition, Competition, And Activation In A Model Of Visually Grounded Speech (2019)0.00
• Language Learning Using Speech To Image Retrieval (2019)9.41
• Syllable Discovery And Cross-lingual Generalization In A Visually Grounded, Self-supervised Speech Model (2023)7.81
• Leveraging Pretrained Image-text Models For Improving Audio-visual Learning (2023)0.00
• Learning Hierarchical Discrete Linguistic Units From Visually-grounded Speech (2019)0.00
• Towards Visually Grounded Sub-word Speech Unit Discovery (2019)9.03
• Representations Of Language In A Model Of Visually Grounded Speech Signal (2017)12.02
• Word Discovery In Visually Grounded, Self-supervised Speech Models (2022)14.08