Residual Memory Networks: Feed-forward Approach To Learn Long Temporal Dependencies

Abstract

Training deep recurrent neural network (RNN) architectures is complicated due to the increased network complexity. This disrupts the learning of higher order abstracts using deep RNN. In case of feed-forward networks training deep structures is simple and faster while learning long-term temporal information is not possible. In this paper we propose a residual memory neural network (RMN) architecture to model short-time dependencies using deep feed-forward layers having residual and time delayed connections. The residual connection paves way to construct deeper networks by enabling unhindered flow of gradients and the time delay units capture temporal information with shared weights. The number of layers in RMN signifies both the hierarchical processing depth and temporal depth. The computational complexity in training RMN is significantly less when compared to deep recurrent networks. RMN is further extended as bi-directional RMN (BRMN) to capture both past and future information. Expe

Stats

• citations8
• S2 citations—
• github stars0
• HF likes0
• heat score7.16
• arxiv keybaskar2018residual

Related papers

• Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate (2023)8.35
• Linear Memory Networks (2018)6.34
• Learning The Sequential Temporal Information With Recurrent Neural Networks (2018)0.00
• Incremental Training Of A Recurrent Neural Network Exploiting A Multi-scale Dynamic Memory (2020)3.58
• On A Novel Training Algorithm For Sequence-to-sequence Predictive Recurrent Networks (2021)0.00
• Restricted Recurrent Neural Networks (2019)7.50
• Persistent Hidden States And Nonlinear Transformation For Long Short-term Memory (2018)8.09
• High Order Recurrent Neural Networks For Acoustic Modelling (2018)8.60