Linear Memory Networks

Abstract

Recurrent neural networks can learn complex transduction problems that require maintaining and actively exploiting a memory of their inputs. Such models traditionally consider memory and input-output functionalities indissolubly entangled. We introduce a novel recurrent architecture based on the conceptual separation between the functional input-output transformation and the memory mechanism, showing how they can be implemented through different neural components. By building on such conceptualization, we introduce the Linear Memory Network, a recurrent model comprising a feedforward neural network, realizing the non-linear functional transformation, and a linear autoencoder for sequences, implementing the memory component. The resulting architecture can be efficiently trained by building on closed-form solutions to linear optimization problems. Further, by exploiting equivalence results between feedforward and recurrent neural networks we devise a pretraining schema for the proposed a

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keybacciu2018linear

Related papers

• Residual Memory Networks: Feed-forward Approach To Learn Long Temporal Dependencies (2018)7.16
• Persistent Hidden States And Nonlinear Transformation For Long Short-term Memory (2018)8.09
• Incremental Training Of A Recurrent Neural Network Exploiting A Multi-scale Dynamic Memory (2020)3.58
• On A Novel Training Algorithm For Sequence-to-sequence Predictive Recurrent Networks (2021)0.00
• From Nodes To Networks: Evolving Recurrent Neural Networks (2018)0.00
• Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate (2023)8.35
• Learning The Sequential Temporal Information With Recurrent Neural Networks (2018)0.00
• Memory Visualization For Gated Recurrent Neural Networks In Speech Recognition (2016)11.76