Speechclip: Integrating Speech With Pre-trained Vision And Language Model

Abstract

Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed speech data is costly. Therefore, we propose SpeechCLIP, a novel framework bridging speech and text through images to enhance speech models without transcriptions. We leverage state-of-the-art pre-trained HuBERT and CLIP, aligning them via paired images and spoken captions with minimal fine-tuning. SpeechCLIP outperforms prior state-of-the-art on image-speech retrieval and performs zero-shot speech-text retrieval without direct supervision from transcriptions. Moreover, SpeechCLIP can directly retrieve semantically related keywords from speech.

Stats

• citations20
• S2 citations—
• github stars0
• HF likes0
• heat score9.92
• arxiv keyshih2022speechclip

Related papers

• Speechclip+: Self-supervised Multi-task Representation Learning For Speech Via CLIP And Speech-image Data (2024)0.00
• Leveraging Pretrained Image-text Models For Improving Audio-visual Learning (2023)0.00
• Clipsonic: Text-to-audio Synthesis With Unlabeled Videos And Pretrained Language-vision Models (2023)9.03
• CLASP: Contrastive Language-speech Pretraining For Multilingual Multimodal Information Retrieval (2024)0.00
• Semantic Speech Retrieval With A Visually Grounded Model Of Untranscribed Speech (2017)10.61
• Language Learning Using Speech To Image Retrieval (2019)9.41
• Clapspeech: Learning Prosody From Text Context With Contrastive Language-audio Pre-training (2023)0.00
• Speechlm: Enhanced Speech Pre-training With Unpaired Textual Data (2022)0.00