Leveraging Pretrained Image-text Models For Improving Audio-visual Learning

Abstract

Visually grounded speech systems learn from paired images and their spoken captions. Recently, there have been attempts to utilize the visually grounded models trained from images and their corresponding text captions, such as CLIP, to improve speech-based visually grounded models' performance. However, the majority of these models only utilize the pretrained image encoder. Cascaded SpeechCLIP attempted to generate localized word-level information and utilize both the pretrained image and text encoders. Despite using both, they noticed a substantial drop in retrieval performance. We proposed Segmental SpeechCLIP which used a hierarchical segmental speech encoder to generate sequences of word-like units. We used the pretrained CLIP text encoder on top of these word-like unit representations and showed significant improvements over the cascaded variant of SpeechCLIP. Segmental SpeechCLIP directly learns the word embeddings as input to the CLIP text encoder bypassing the vocabulary embedd

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybhati2023leveraging

Related papers

• Speechclip: Integrating Speech With Pre-trained Vision And Language Model (2022)9.92
• Speechclip+: Self-supervised Multi-task Representation Learning For Speech Via CLIP And Speech-image Data (2024)0.00
• Clipsonic: Text-to-audio Synthesis With Unlabeled Videos And Pretrained Language-vision Models (2023)9.03
• Large-scale Representation Learning From Visually Grounded Untranscribed Speech (2019)10.48
• Language Learning Using Speech To Image Retrieval (2019)9.41
• On The Language Encoder Of Contrastive Cross-modal Models (2023)0.00
• Cross-modal Audio-visual Co-learning For Text-independent Speaker Verification (2023)9.23
• Learning Audio-video Modalities From Image Captions (2022)12.54