Persistent Hidden States And Nonlinear Transformation For Long Short-term Memory

Abstract

Recurrent neural networks (RNNs) have been drawing much attention with great success in many applications like speech recognition and neural machine translation. Long short-term memory (LSTM) is one of the most popular RNN units in deep learning applications. LSTM transforms the input and the previous hidden states to the next states with the affine transformation, multiplication operations and a nonlinear activation function, which makes a good data representation for a given task. The affine transformation includes rotation and reflection, which change the semantic or syntactic information of dimensions in the hidden states. However, considering that a model interprets the output sequence of LSTM over the whole input sequence, the dimensions of the states need to keep the same type of semantic or syntactic information regardless of the location in the sequence. In this paper, we propose a simple variant of the LSTM unit, persistent recurrent unit (PRU), where each dimension of hidden

Stats

• citations11
• S2 citations—
• github stars0
• HF likes0
• heat score8.09
• arxiv keychoi2018persistent

Related papers

• Memory Visualization For Gated Recurrent Neural Networks In Speech Recognition (2016)11.76
• Linear Memory Networks (2018)6.34
• Restricted Recurrent Neural Networks (2019)7.50
• From Nodes To Networks: Evolving Recurrent Neural Networks (2018)0.00
• Residual Memory Networks: Feed-forward Approach To Learn Long Temporal Dependencies (2018)7.16
• Bidirectional Quaternion Long-short Term Memory Recurrent Neural Networks For Speech Recognition (2018)9.41
• High Order Recurrent Neural Networks For Acoustic Modelling (2018)8.60
• Learning Compact Recurrent Neural Networks (2016)0.00