Sequence Training Of DNN Acoustic Models With Natural Gradient

Abstract

Deep Neural Network (DNN) acoustic models often use discriminative sequence training that optimises an objective function that better approximates the word error rate (WER) than frame-based training. Sequence training is normally implemented using Stochastic Gradient Descent (SGD) or Hessian Free (HF) training. This paper proposes an alternative batch style optimisation framework that employs a Natural Gradient (NG) approach to traverse through the parameter space. By correcting the gradient according to the local curvature of the KL-divergence, the NG optimisation process converges more quickly than HF. Furthermore, the proposed NG approach can be applied to any sequence discriminative training criterion. The efficacy of the NG method is shown using experiments on a Multi-Genre Broadcast (MGB) transcription task that demonstrates both the computational efficiency and the accuracy of the resulting DNN models.

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keyhaider2018sequence

Related papers

• A Distributed Optimisation Framework Combining Natural Gradient With Hessian-free For Discriminative Sequence Training (2021)0.00
• Distributed Training Of Deep Neural Network Acoustic Models For Automatic Speech Recognition (2020)0.00
• Sequence Training And Adaptation Of Highway Deep Neural Networks (2016)3.58
• Low-rank And Sparse Soft Targets To Learn Better DNN Acoustic Models (2016)3.58
• High Order Recurrent Neural Networks For Acoustic Modelling (2018)8.60
• Dynamic Layer Normalization For Adaptive Neural Acoustic Modeling In Speech Recognition (2017)11.58
• Improving Trajectory Modelling For Dnn-based Speech Synthesis By Using Stacked Bottleneck Features And Minimum Generation Error Training (2016)10.35
• Batch-normalized Joint Training For Dnn-based Distant Speech Recognition (2017)8.82