Generating Adjacency-constrained Subgoals In Hierarchical Reinforcement Learning

Abstract

Goal-conditioned hierarchical reinforcement learning (HRL) is a promising approach for scaling up reinforcement learning (RL) techniques. However, it often suffers from training inefficiency as the action space of the high-level, i.e., the goal space, is often large. Searching in a large goal space poses difficulties for both high-level subgoal generation and low-level policy learning. In this paper, we show that this problem can be effectively alleviated by restricting the high-level action space from the whole goal space to a \(k\)-step adjacent region of the current state using an adjacency constraint. We theoretically prove that the proposed adjacency constraint preserves the optimal hierarchical policy in deterministic MDPs, and show that this constraint can be practically implemented by training an adjacency network that can discriminate between adjacent and non-adjacent subgoals. Experimental results on discrete and continuous control tasks show that incorporating the adjacency

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2020generating

Related papers

• State-conditioned Adversarial Subgoal Generation (2022)0.00
• Learning Representations In Model-free Hierarchical Reinforcement Learning (2018)11.49
• Subgoal-based Hierarchical Reinforcement Learning For Multi-agent Collaboration (2024)0.00
• Goal Space Abstraction In Hierarchical Reinforcement Learning Via Reachability Analysis (2023)0.00
• MENTOR: Guiding Hierarchical Reinforcement Learning With Human Feedback And Dynamic Distance Constraint (2024)6.34
• Learning And Exploiting Multiple Subgoals For Fast Exploration In Hierarchical Reinforcement Learning (2019)0.00
• Guided Cooperation In Hierarchical Reinforcement Learning Via Model-based Rollout (2023)0.00
• Bidirectional-reachable Hierarchical Reinforcement Learning With Mutually Responsive Policies (2024)0.00