Reinforcement Learning With Brain-inspired Modulation Can Improve Adaptation To Environmental Changes

Abstract

Developments in reinforcement learning (RL) have allowed algorithms to achieve impressive performance in highly complex, but largely static problems. In contrast, biological learning seems to value efficiency of adaptation to a constantly-changing world. Here we build on a recently-proposed neuronal learning rule that assumes each neuron can optimize its energy balance by predicting its own future activity. That assumption leads to a neuronal learning rule that uses presynaptic input to modulate prediction error. We argue that an analogous RL rule would use action probability to modulate reward prediction error. This modulation makes the agent more sensitive to negative experiences, and more careful in forming preferences. We embed the proposed rule in both tabular and deep-Q-network RL algorithms, and find that it outperforms conventional algorithms in simple, but highly-dynamic tasks. We suggest that the new rule encapsulates a core principle of biological intelligence; an important

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychalmers2022reinforcement

Related papers

• Context Meta-reinforcement Learning Via Neuromodulation (2021)6.34
• Lifelong Reinforcement Learning Via Neuromodulation (2024)0.00
• Noise-based Reward-modulated Learning (2025)0.00
• Efficient Off-policy Reinforcement Learning Via Brain-inspired Computing (2022)8.35
• Partial Models For Building Adaptive Model-based Reinforcement Learning Agents (2024)0.00
• Learning Fast Changing Slow In Spiking Neural Networks (2024)3.58
• An Introduction To Reinforcement Learning For Neuroscience (2023)5.84
• On The Reliability And Generalizability Of Brain-inspired Reinforcement Learning Algorithms (2020)0.00