Fast-slow Transformer For Visually Grounding Speech

Abstract

We present Fast-Slow Transformer for Visually Grounding Speech, or FaST-VGS. FaST-VGS is a Transformer-based model for learning the associations between raw speech waveforms and visual images. The model unifies dual-encoder and cross-attention architectures into a single model, reaping the superior retrieval speed of the former along with the accuracy of the latter. FaST-VGS achieves state-of-the-art speech-image retrieval accuracy on benchmark datasets, and its learned representations exhibit strong performance on the ZeroSpeech 2021 phonetic and semantic tasks.

Stats

• citations19
• S2 citations—
• github stars0
• HF likes0
• heat score9.76
• arxiv keypeng2021fast

Related papers

• Self-supervised Representation Learning For Speech Using Visual Grounding And Masked Language Modeling (2022)0.00
• SSAST: Self-supervised Audio Spectrogram Transformer (2021)17.61
• Fastvc: Fast Voice Conversion With Non-parallel Data (2020)5.24
• Daspeech: Directed Acyclic Transformer For Fast And High-quality Speech-to-speech Translation (2023)5.24
• Fastfit: Towards Real-time Iterative Neural Vocoder By Replacing U-net Encoder With Multiple Stfts (2023)0.00
• Fastspeech: Fast, Robust And Controllable Text To Speech (2019)0.00
• Self-supervised Rewiring Of Pre-trained Speech Encoders: Towards Faster Fine-tuning With Less Labels In Speech Processing (2022)3.58
• Study Of Lightweight Transformer Architectures For Single-channel Speech Enhancement (2025)3.58