Sequence Training And Adaptation Of Highway Deep Neural Networks

Abstract

Highway deep neural network (HDNN) is a type of depth-gated feedforward neural network, which has shown to be easier to train with more hidden layers and also generalise better compared to conventional plain deep neural networks (DNNs). Previously, we investigated a structured HDNN architecture for speech recognition, in which the two gate functions were tied across all the hidden layers, and we were able to train a much smaller model without sacrificing the recognition accuracy. In this paper, we carry on the study of this architecture with sequence-discriminative training criterion and speaker adaptation techniques on the AMI meeting speech recognition corpus. We show that these two techniques improve speech recognition accuracy on top of the model trained with the cross entropy criterion. Furthermore, we demonstrate that the two gate functions that are tied across all the hidden layers are able to control the information flow over the whole network, and we can achieve considerable i

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keylu2016sequence

Related papers

• Knowledge Distillation For Small-footprint Highway Networks (2016)12.25
• Sequence Training Of DNN Acoustic Models With Natural Gradient (2018)5.24
• Bayesian Learning For Deep Neural Network Adaptation (2020)9.76
• A Comparison Of Adaptation Techniques And Recurrent Neural Network Architectures (2018)3.58
• Hypertts: Parameter Efficient Adaptation In Text To Speech Using Hypernetworks (2024)3.23
• Embeddings For DNN Speaker Adaptive Training (2019)7.16
• End-to-end Attention-based Distant Speech Recognition With Highway LSTM (2016)0.00
• Language Modeling With Highway LSTM (2017)10.21