From Nodes To Networks: Evolving Recurrent Neural Networks

Abstract

Gated recurrent networks such as those composed of Long Short-Term Memory (LSTM) nodes have recently been used to improve state of the art in many sequential processing tasks such as speech recognition and machine translation. However, the basic structure of the LSTM node is essentially the same as when it was first conceived 25 years ago. Recently, evolutionary and reinforcement learning mechanisms have been employed to create new variations of this structure. This paper proposes a new method, evolution of a tree-based encoding of the gated memory nodes, and shows that it makes it possible to explore new variations more effectively than other methods. The method discovers nodes with multiple recurrent paths and multiple memory cells, which lead to significant improvement in the standard language modeling benchmark task. The paper also shows how the search process can be speeded up by training an LSTM network to estimate performance of candidate structures, and by encouraging explorati

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyrawal2018from

Related papers

• Memory Visualization For Gated Recurrent Neural Networks In Speech Recognition (2016)11.76
• Improving Speech Recognition By Revising Gated Recurrent Units (2017)11.19
• Investigating Gated Recurrent Neural Networks For Speech Synthesis (2016)0.00
• Learning The Sequential Temporal Information With Recurrent Neural Networks (2018)0.00
• Understanding Recurrent Neural State Using Memory Signatures (2018)0.00
• A Comparison Of Adaptation Techniques And Recurrent Neural Network Architectures (2018)3.58
• Light Gated Recurrent Units For Speech Recognition (2018)18.90
• Persistent Hidden States And Nonlinear Transformation For Long Short-term Memory (2018)8.09