Reusable Options Through Gradient-based Meta Learning

Abstract

Hierarchical methods in reinforcement learning have the potential to reduce the amount of decisions that the agent needs to perform when learning new tasks. However, finding reusable useful temporal abstractions that facilitate fast learning remains a challenging problem. Recently, several deep learning approaches were proposed to learn such temporal abstractions in the form of options in an end-to-end manner. In this work, we point out several shortcomings of these methods and discuss their potential negative consequences. Subsequently, we formulate the desiderata for reusable options and use these to frame the problem of learning options as a gradient-based meta-learning problem. This allows us to formulate an objective that explicitly incentivizes options which allow a higher-level decision maker to adjust in few steps to different tasks. Experimentally, we show that our method is able to learn transferable components which accelerate learning and performs better than existing prior

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykuric2022reusable

Related papers

• A Provably Efficient Option-based Algorithm For Both High-level And Low-level Learning (2024)0.00
• Multi-timescale, Gradient Descent, Temporal Difference Learning With Linear Options (2017)0.00
• Hyperbolic Embeddings For Learning Options In Hierarchical Reinforcement Learning (2018)0.00
• Autonomous Option Invention For Continual Hierarchical Reinforcement Learning And Planning (2024)2.26
• Optimal Options For Multi-task Reinforcement Learning Under Time Constraints (2020)0.00
• Boosting Hierarchical Reinforcement Learning With Meta-learning For Complex Task Adaptation (2024)0.00
• Discovering Hierarchies Using Imitation Learning From Hierarchy Aware Policies (2018)0.00
• Learning And Exploiting Multiple Subgoals For Fast Exploration In Hierarchical Reinforcement Learning (2019)0.00