Distributed Training Of Deep Neural Network Acoustic Models For Automatic Speech Recognition

Abstract

The past decade has witnessed great progress in Automatic Speech Recognition (ASR) due to advances in deep learning. The improvements in performance can be attributed to both improved models and large-scale training data. Key to training such models is the employment of efficient distributed learning techniques. In this article, we provide an overview of distributed training techniques for deep neural network acoustic models for ASR. Starting with the fundamentals of data parallel stochastic gradient descent (SGD) and ASR acoustic modeling, we will investigate various distributed training strategies and their realizations in high performance computing (HPC) environments with an emphasis on striking the balance between communication and computation. Experiments are carried out on a popular public benchmark to study the convergence, speedup and recognition performance of the investigated strategies.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keycui2020distributed

Related papers

• A Network Of Deep Neural Networks For Distant Speech Recognition (2017)10.35
• Automatic Speech Recognition Using Advanced Deep Learning Approaches: A Survey (2024)16.63
• Bigssl: Exploring The Frontier Of Large-scale Semi-supervised Learning For Automatic Speech Recognition (2021)15.73
• Deep Learning For Distant Speech Recognition (2017)0.00
• Exponential Moving Average Model In Parallel Speech Recognition Training (2017)0.00
• A Method To Reveal Speaker Identity In Distributed ASR Training, And How To Counter It (2021)5.84
• Sequence Training Of DNN Acoustic Models With Natural Gradient (2018)5.24
• Ensemble Of Jointly Trained Deep Neural Network-based Acoustic Models For Reverberant Speech Recognition (2016)0.00