Enabling Differentially Private Federated Learning For Speech Recognition: Benchmarks, Adaptive Optimizers And Gradient Clipping

Abstract

While federated learning (FL) and differential privacy (DP) have been extensively studied, their application to automatic speech recognition (ASR) remains largely unexplored due to the challenges in training large transformer models. Specifically, large models further exacerbate issues in FL as they are particularly susceptible to gradient heterogeneity across layers, unlike the relatively uniform gradient behavior observed in shallow models. As a result, prior works struggle to converge with standard optimization techniques, even in the absence of DP mechanisms. To the best of our knowledge, no existing work establishes a competitive, practical recipe for FL with DP in the context of ASR. To address this gap, we establish \textbf\{the first benchmark for FL with DP in end-to-end ASR\}. Our approach centers on per-layer clipping and layer-wise gradient normalization: theoretical analysis reveals that these techniques together mitigate clipping bias and gradient heterogeneity across lay

Stats

• citations0
• S2 citations—
• github stars18
• HF likes0
• heat score2.56
• arxiv keypelikan2023enabling

Related papers

• Importance Of Smoothness Induced By Optimizers In FL4ASR: Towards Understanding Federated Learning For End-to-end ASR (2023)0.00
• Communication-efficient Personalized Federated Learning For Speech-to-text Tasks (2024)7.81
• Automatic Speech Recognition Using Advanced Deep Learning Approaches: A Survey (2024)16.63
• Private Language Model Adaptation For Speech Recognition (2021)0.00
• A Method To Reveal Speaker Identity In Distributed ASR Training, And How To Counter It (2021)5.84
• Efficiently Train ASR Models That Memorize Less And Perform Better With Per-core Clipping (2024)0.00
• Revisit Micro-batch Clipping: Adaptive Data Pruning Via Gradient Manipulation (2024)0.00
• The Gift Of Feedback: Improving ASR Model Quality By Learning From User Corrections Through Federated Learning (2023)0.00