Guided Cooperation In Hierarchical Reinforcement Learning Via Model-based Rollout

Abstract

Goal-conditioned hierarchical reinforcement learning (HRL) presents a promising approach for enabling effective exploration in complex, long-horizon reinforcement learning (RL) tasks through temporal abstraction. Empirically, heightened inter-level communication and coordination can induce more stable and robust policy improvement in hierarchical systems. Yet, most existing goal-conditioned HRL algorithms have primarily focused on the subgoal discovery, neglecting inter-level cooperation. Here, we propose a goal-conditioned HRL framework named Guided Cooperation via Model-based Rollout (GCMR), aiming to bridge inter-layer information synchronization and cooperation by exploiting forward dynamics. Firstly, the GCMR mitigates the state-transition error within off-policy correction via model-based rollout, thereby enhancing sample efficiency. Secondly, to prevent disruption by the unseen subgoals and states, lower-level Q-function gradients are constrained using a gradient penalty with a

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2023guided

Related papers

• Subgoal-based Hierarchical Reinforcement Learning For Multi-agent Collaboration (2024)0.00
• Bidirectional-reachable Hierarchical Reinforcement Learning With Mutually Responsive Policies (2024)0.00
• Generating Adjacency-constrained Subgoals In Hierarchical Reinforcement Learning (2020)0.00
• MENTOR: Guiding Hierarchical Reinforcement Learning With Human Feedback And Dynamic Distance Constraint (2024)6.34
• Exploring The Limits Of Hierarchical World Models In Reinforcement Learning (2024)6.34
• Learning Representations In Model-free Hierarchical Reinforcement Learning (2018)11.49
• Hierarchical Reinforcement Learning With Optimal Level Synchronization Based On A Deep Generative Model (2021)0.00
• State-conditioned Adversarial Subgoal Generation (2022)0.00