Semantic Speech Retrieval With A Visually Grounded Model Of Untranscribed Speech

Abstract

There is growing interest in models that can learn from unlabelled speech paired with visual context. This setting is relevant for low-resource speech processing, robotics, and human language acquisition research. Here we study how a visually grounded speech model, trained on images of scenes paired with spoken captions, captures aspects of semantics. We use an external image tagger to generate soft text labels from images, which serve as targets for a neural model that maps untranscribed speech to (semantic) keyword labels. We introduce a newly collected data set of human semantic relevance judgements and an associated task, semantic speech retrieval, where the goal is to search for spoken utterances that are semantically relevant to a given text query. Without seeing any text, the model trained on parallel speech and images achieves a precision of almost 60% on its top ten semantic retrievals. Compared to a supervised model trained on transcriptions, our model matches human judgement

Stats

• citations25
• S2 citations—
• github stars0
• HF likes0
• heat score10.61
• arxiv keykamper2017semantic

Related papers

• On The Contributions Of Visual And Textual Supervision In Low-resource Semantic Speech Retrieval (2019)6.34
• Semantic Query-by-example Speech Search Using Visual Grounding (2019)7.81
• Language Learning Using Speech To Image Retrieval (2019)9.41
• Large-scale Representation Learning From Visually Grounded Untranscribed Speech (2019)10.48
• Keyword Localisation In Untranscribed Speech Using Visually Grounded Speech Models (2022)6.34
• Symbolic Inductive Bias For Visually Grounded Learning Of Spoken Language (2018)5.24
• Representations Of Language In A Model Of Visually Grounded Speech Signal (2017)12.02
• Word Discovery In Visually Grounded, Self-supervised Speech Models (2022)14.08