Augmenting Images For ASR And TTS Through Single-loop And Dual-loop Multimodal Chain Framework

Abstract

Previous research has proposed a machine speech chain to enable automatic speech recognition (ASR) and text-to-speech synthesis (TTS) to assist each other in semi-supervised learning and to avoid the need for a large amount of paired speech and text data. However, that framework still requires a large amount of unpaired (speech or text) data. A prototype multimodal machine chain was then explored to further reduce the need for a large amount of unpaired data, which could improve ASR or TTS even when no more speech or text data were available. Unfortunately, this framework relied on the image retrieval (IR) model, and thus it was limited to handling only those images that were already known during training. Furthermore, the performance of this framework was only investigated with single-speaker artificial speech data. In this study, we revamp the multimodal machine chain framework with image generation (IG) and investigate the possibility of augmenting image data for ASR and TTS using s

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyeffendi2020augmenting

Related papers

• Listening While Speaking And Visualizing: Improving ASR Through Multimodal Chain (2019)4.52
• Machine Speech Chain With One-shot Speaker Adaptation (2018)11.29
• Exploring Machine Speech Chain For Domain Adaptation And Few-shot Speaker Adaptation (2021)0.00
• Improving Multimodal Speech Recognition By Data Augmentation And Speech Representations (2022)9.03
• Tokenchain: A Discrete Speech Chain Via Semantic Token Modeling (2025)0.00
• Multimodal Grounding For Sequence-to-sequence Speech Recognition (2018)8.82
• Improving Low Resource Code-switched ASR Using Augmented Code-switched TTS (2020)7.50
• ASR Data Augmentation In Low-resource Settings Using Cross-lingual Multi-speaker TTS And Cross-lingual Voice Conversion (2022)6.77