Reservoir Computing For Fast, Simplified Reinforcement Learning On Memory Tasks

Abstract

Tasks in which rewards depend upon past information not available in the current observation set can only be solved by agents that are equipped with short-term memory. Usual choices for memory modules include trainable recurrent hidden layers, often with gated memory. Reservoir computing presents an alternative, in which a recurrent layer is not trained, but rather has a set of fixed, sparse recurrent weights. The weights are scaled to produce stable dynamical behavior such that the reservoir state contains a high-dimensional, nonlinear impulse response function of the inputs. An output decoder network can then be used to map the compressive history represented by the reservoir's state to any outputs, including agent actions or predictions. In this study, we find that reservoir computing greatly simplifies and speeds up reinforcement learning on memory tasks by (1) eliminating the need for backpropagation of gradients through time, (2) presenting all recent history simultaneously to th

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymckee2024reservoir

Related papers

• Low-pass Recurrent Neural Networks - A Memory Architecture For Longer-term Correlation Discovery (2018)0.00
• Efficient Quantum Recurrent Reinforcement Learning Via Quantum Reservoir Computing (2023)0.00
• Integrating Episodic Memory Into A Reinforcement Learning Agent Using Reservoir Sampling (2018)0.00
• Stable Hadamard Memory: Revitalizing Memory-augmented Agents For Reinforcement Learning (2024)0.00
• Influence-aware Memory Architectures For Deep Reinforcement Learning (2019)2.26
• Map-based Experience Replay: A Memory-efficient Solution To Catastrophic Forgetting In Reinforcement Learning (2023)9.23
• Reward Function Compression Facilitates Goal-dependent Reinforcement Learning (2025)0.00
• Augmented Replay Memory In Reinforcement Learning With Continuous Control (2019)5.24