Federated Marginal Personalization For ASR Rescoring

Abstract

We introduce federated marginal personalization (FMP), a novel method for continuously updating personalized neural network language models (NNLMs) on private devices using federated learning (FL). Instead of fine-tuning the parameters of NNLMs on personal data, FMP regularly estimates global and personalized marginal distributions of words, and adjusts the probabilities from NNLMs by an adaptation factor that is specific to each word. Our presented approach can overcome the limitations of federated fine-tuning and efficiently learn personalized NNLMs on devices. We study the application of FMP on second-pass ASR rescoring tasks. Experiments on two speech evaluation datasets show modest word error rate (WER) reductions. We also demonstrate that FMP could offer reasonable privacy with only a negligible cost in speech recognition accuracy.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyliu2020federated

Related papers

• Private Language Model Adaptation For Speech Recognition (2021)0.00
• The Gift Of Feedback: Improving ASR Model Quality By Learning From User Corrections Through Federated Learning (2023)0.00
• Fast Contextual Adaptation With Neural Associative Memory For On-device Personalized Speech Recognition (2021)9.76
• Communication-efficient Personalized Federated Learning For Speech-to-text Tasks (2024)7.81
• Personalization For Bert-based Discriminative Speech Recognition Rescoring (2023)5.24
• Privacy Attacks For Automatic Speech Recognition Acoustic Models In A Federated Learning Framework (2021)9.23
• The Universal Personalizer: Few-shot Dysarthric Speech Recognition Via Meta-learning (2025)0.00
• Deep Shallow Fusion For RNN-T Personalization (2020)12.81