Globally Optimal Hierarchical Reinforcement Learning For Linearly-solvable Markov Decision Processes

Abstract

In this work we present a novel approach to hierarchical reinforcement learning for linearly-solvable Markov decision processes. Our approach assumes that the state space is partitioned, and the subtasks consist in moving between the partitions. We represent value functions on several levels of abstraction, and use the compositionality of subtasks to estimate the optimal values of the states in each partition. The policy is implicitly defined on these optimal value estimates, rather than being decomposed among the subtasks. As a consequence, our approach can learn the globally optimal policy, and does not suffer from the non-stationarity of high-level decisions. If several partitions have equivalent dynamics, the subtasks of those partitions can be shared. If the set of boundary states is smaller than the entire state space, our approach can have significantly smaller sample complexity than that of a flat learner, and we validate this empirically in several experiments.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyinfante2021globally

Related papers

• Hierarchy Through Composition With Linearly Solvable Markov Decision Processes (2016)0.00
• Deep Hierarchical Reinforcement Learning Algorithm In Partially Observable Markov Decision Processes (2018)12.87
• Intrinsically Motivated Hierarchical Policy Learning In Multi-objective Markov Decision Processes (2023)4.52
• Hyperbolic Embeddings For Learning Options In Hierarchical Reinforcement Learning (2018)0.00
• Near-optimal Representation Learning For Hierarchical Reinforcement Learning (2018)0.00
• Nearly Minimax Optimal Reinforcement Learning For Linear Markov Decision Processes (2022)0.00
• Performative Reinforcement Learning With Linear Markov Decision Process (2024)0.00
• Latent Space Policies For Hierarchical Reinforcement Learning (2018)0.00