Self-organization Of Action Hierarchy And Compositionality By Reinforcement Learning With Recurrent Neural Networks

Abstract

Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown distinct advantages, e.g., solving memory-dependent tasks and meta-learning. However, little effort has been spent on improving RNN architectures and on understanding the underlying neural mechanisms for performance gain. In this paper, we propose a novel, multiple-timescale, stochastic RNN for RL. Empirical results show that the network can autonomously learn to abstract sub-goals and can self-develop an action hierarchy using internal dynamics in a challenging continuous control task. Furthermore, we show that the self-developed compositionality of the network enhances faster re-learning when adapting to a new task that is a re-composition of previously learned sub-goals, than when starting from scratch. We also found that improved performance can be achieved when neural activities are subject to stochastic rather than deterministic dynamics.

Stats

• citations13
• S2 citations—
• github stars0
• HF likes0
• heat score8.60
• arxiv keyhan2019self

Related papers

• Emergent Temporal Abstractions In Autoregressive Models Enable Hierarchical Reinforcement Learning (2025)0.00
• Dynamic Deep-reinforcement-learning Algorithm In Partially Observable Markov Decision Processes (2023)0.00
• Learning Representations In Model-free Hierarchical Reinforcement Learning (2018)11.49
• Verifiable And Compositional Reinforcement Learning Systems (2021)0.00
• Learning To Reinforcement Learn (2016)0.00
• Subgoal-based Hierarchical Reinforcement Learning For Multi-agent Collaboration (2024)0.00
• Multi-horizon Representations With Hierarchical Forward Models For Reinforcement Learning (2022)0.00
• RLOC: Neurobiologically Inspired Hierarchical Reinforcement Learning Algorithm For Continuous Control Of Nonlinear Dynamical Systems (2019)0.00