Fedspeech: Federated Text-to-speech With Continual Learning

Abstract

Federated learning enables collaborative training of machine learning models under strict privacy restrictions and federated text-to-speech aims to synthesize natural speech of multiple users with a few audio training samples stored in their devices locally. However, federated text-to-speech faces several challenges: very few training samples from each speaker are available, training samples are all stored in local device of each user, and global model is vulnerable to various attacks. In this paper, we propose a novel federated learning architecture based on continual learning approaches to overcome the difficulties above. Specifically, 1) we use gradual pruning masks to isolate parameters for preserving speakers' tones; 2) we apply selective masks for effectively reusing knowledge from tasks; 3) a private speaker embedding is introduced to keep users' privacy. Experiments on a reduced VCTK dataset demonstrate the effectiveness of FedSpeech: it nearly matches multi-task training in te

Stats

• citations16
• S2 citations—
• github stars0
• HF likes0
• heat score9.23
• arxiv keyjiang2021fedspeech

Related papers

• Communication-efficient Personalized Federated Learning For Speech-to-text Tasks (2024)7.81
• Federated Pruning: Improving Neural Network Efficiency With Federated Learning (2022)7.50
• Fednst: Federated Noisy Student Training For Automatic Speech Recognition (2022)6.77
• Training Speech Recognition Models With Federated Learning: A Quality/cost Framework (2020)12.93
• Federated Learning For Keyword Spotting (2018)17.09
• Fed-pisa: Federated Voice Cloning Via Personalized Identity-style Adaptation (2025)0.00
• Continual Speaker Adaptation For Text-to-speech Synthesis (2021)0.00
• Separate But Together: Unsupervised Federated Learning For Speech Enhancement From Non-iid Data (2021)8.35