Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate

Abstract

Recurrent Neural Networks (RNNs) are widely recognized for their proficiency in modeling temporal dependencies, making them highly prevalent in sequential data processing applications. Nevertheless, vanilla RNNs are confronted with the well-known issue of gradient vanishing and exploding, posing a significant challenge for learning and establishing long-range dependencies. Additionally, gated RNNs tend to be over-parameterized, resulting in poor computational efficiency and network generalization. To address these challenges, this paper proposes a novel Delayed Memory Unit (DMU). The DMU incorporates a delay line structure along with delay gates into vanilla RNN, thereby enhancing temporal interaction and facilitating temporal credit assignment. Specifically, the DMU is designed to directly distribute the input information to the optimal time instant in the future, rather than aggregating and redistributing it over time through intricate network dynamics. Our proposed DMU demonstrates

Stats

• citations12
• S2 citations—
• github stars0
• HF likes0
• heat score8.35
• arxiv keysun2023delayed

Related papers

• Residual Memory Networks: Feed-forward Approach To Learn Long Temporal Dependencies (2018)7.16
• Memory Visualization For Gated Recurrent Neural Networks In Speech Recognition (2016)11.76
• Gated Recurrent Unit Based Acoustic Modeling With Future Context (2018)7.16
• Learning The Sequential Temporal Information With Recurrent Neural Networks (2018)0.00
• Incremental Training Of A Recurrent Neural Network Exploiting A Multi-scale Dynamic Memory (2020)3.58
• Dynamic Gated Recurrent Neural Network For Compute-efficient Speech Enhancement (2024)8.35
• Light Gated Recurrent Units For Speech Recognition (2018)18.90
• Persistent Hidden States And Nonlinear Transformation For Long Short-term Memory (2018)8.09