Incremental Training Of A Recurrent Neural Network Exploiting A Multi-scale Dynamic Memory

Abstract

The effectiveness of recurrent neural networks can be largely influenced by their ability to store into their dynamical memory information extracted from input sequences at different frequencies and timescales. Such a feature can be introduced into a neural architecture by an appropriate modularization of the dynamic memory. In this paper we propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning. First, we show how to extend the architecture of a simple RNN by separating its hidden state into different modules, each subsampling the network hidden activations at different frequencies. Then, we discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies. Each new module works at a slower frequency than the previous ones and it is initialized to encode the subsampled sequence of hidden activations. Experimental results on synthetic and real-world datasets on speech recognition

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keycarta2020incremental

Related papers

• Memory Visualization For Gated Recurrent Neural Networks In Speech Recognition (2016)11.76
• Learning Compact Recurrent Neural Networks (2016)0.00
• Residual Memory Networks: Feed-forward Approach To Learn Long Temporal Dependencies (2018)7.16
• High Order Recurrent Neural Networks For Acoustic Modelling (2018)8.60
• Learning The Sequential Temporal Information With Recurrent Neural Networks (2018)0.00
• Dynamic Gated Recurrent Neural Network For Compute-efficient Speech Enhancement (2024)8.35
• Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate (2023)8.35
• Monaural Speech Enhancement Using A Multi-branch Temporal Convolutional Network (2019)3.58