Near-optimal Representation Learning For Hierarchical Reinforcement Learning

Abstract

We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning. In such hierarchical structures, a higher-level controller solves tasks by iteratively communicating goals which a lower-level policy is trained to reach. Accordingly, the choice of representation -- the mapping of observation space to goal space -- is crucial. To study this problem, we develop a notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation. We derive expressions which bound the sub-optimality and show how these expressions can be translated to representation learning objectives which may be optimized in practice. Results on a number of difficult continuous-control tasks show that our approach to representation learning yields qualitatively better representations as well as quantitatively better hierarchical policies, compared to existing methods (see videos at https://sites.google.com/

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keynachum2018near

Related papers

• Learning Representations In Model-free Hierarchical Reinforcement Learning (2018)11.49
• Multi-horizon Representations With Hierarchical Forward Models For Reinforcement Learning (2022)0.00
• Hyperbolic Embeddings For Learning Options In Hierarchical Reinforcement Learning (2018)0.00
• Generating Adjacency-constrained Subgoals In Hierarchical Reinforcement Learning (2020)0.00
• State-conditioned Adversarial Subgoal Generation (2022)0.00
• Goal Space Abstraction In Hierarchical Reinforcement Learning Via Reachability Analysis (2023)0.00
• Reconciling Spatial And Temporal Abstractions For Goal Representation (2024)1.69
• Globally Optimal Hierarchical Reinforcement Learning For Linearly-solvable Markov Decision Processes (2021)2.26