Text-free Image-to-speech Synthesis Using Learned Segmental Units

Abstract

In this paper we present the first model for directly synthesizing fluent, natural-sounding spoken audio captions for images that does not require natural language text as an intermediate representation or source of supervision. Instead, we connect the image captioning module and the speech synthesis module with a set of discrete, sub-word speech units that are discovered with a self-supervised visual grounding task. We conduct experiments on the Flickr8k spoken caption dataset in addition to a novel corpus of spoken audio captions collected for the popular MSCOCO dataset, demonstrating that our generated captions also capture diverse visual semantics of the images they describe. We investigate several different intermediate speech representations, and empirically find that the representation must satisfy several important properties to serve as drop-in replacements for text.

Stats

• citations27
• S2 citations—
• github stars0
• HF likes0
• heat score10.85
• arxiv keyhsu2020text

Related papers

• Learning Word-like Units From Joint Audio-visual Analysis (2017)12.33
• Large-scale Representation Learning From Visually Grounded Untranscribed Speech (2019)10.48
• Leveraging Pretrained Image-text Models For Improving Audio-visual Learning (2023)0.00
• Jointly Discovering Visual Objects And Spoken Words From Raw Sensory Input (2018)14.27
• Towards Practical And Efficient Image-to-speech Captioning With Vision-language Pre-training And Multi-modal Tokens (2023)6.34
• Show And Speak: Directly Synthesize Spoken Description Of Images (2020)5.24
• SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned To MSCOCO Data Set (2017)8.82
• Seeing What You Say: Expressive Image Generation From Speech (2025)0.00