Federated Pruning: Improving Neural Network Efficiency With Federated Learning

Abstract

Automatic Speech Recognition models require large amount of speech data for training, and the collection of such data often leads to privacy concerns. Federated learning has been widely used and is considered to be an effective decentralized technique by collaboratively learning a shared prediction model while keeping the data local on different clients devices. However, the limited computation and communication resources on clients devices present practical difficulties for large models. To overcome such challenges, we propose Federated Pruning to train a reduced model under the federated setting, while maintaining similar performance compared to the full model. Moreover, the vast amount of clients data can also be leveraged to improve the pruning results compared to centralized training. We explore different pruning schemes and provide empirical evidence of the effectiveness of our methods.

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keylin2022federated

Related papers

• Fedspeech: Federated Text-to-speech With Continual Learning (2021)9.23
• Accurate And Structured Pruning For Efficient Automatic Speech Recognition (2023)7.81
• Training Speech Recognition Models With Federated Learning: A Quality/cost Framework (2020)12.93
• Structured Pruning Of Self-supervised Pre-trained Models For Speech Recognition And Understanding (2023)11.39
• Fednst: Federated Noisy Student Training For Automatic Speech Recognition (2022)6.77
• Private Language Model Adaptation For Speech Recognition (2021)0.00
• The Gift Of Feedback: Improving ASR Model Quality By Learning From User Corrections Through Federated Learning (2023)0.00
• Dynamic Sparsity Neural Networks For Automatic Speech Recognition (2020)0.00