Autonomous Option Invention For Continual Hierarchical Reinforcement Learning And Planning

Abstract

Abstraction is key to scaling up reinforcement learning (RL). However, autonomously learning abstract state and action representations to enable transfer and generalization remains a challenging open problem. This paper presents a novel approach for inventing, representing, and utilizing options, which represent temporally extended behaviors, in continual RL settings. Our approach addresses streams of stochastic problems characterized by long horizons, sparse rewards, and unknown transition and reward functions. Our approach continually learns and maintains an interpretable state abstraction, and uses it to invent high-level options with abstract symbolic representations. These options meet three key desiderata: (1) composability for solving tasks effectively with lookahead planning, (2) reusability across problem instances for minimizing the need for relearning, and (3) mutual independence for reducing interference among options. Our main contributions are approaches for continually

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keynayyar2024autonomous

Related papers

• Multi-timescale, Gradient Descent, Temporal Difference Learning With Linear Options (2017)0.00
• A Provably Efficient Option-based Algorithm For Both High-level And Low-level Learning (2024)0.00
• A Theory Of Abstraction In Reinforcement Learning (2022)0.00
• Emergent Temporal Abstractions In Autoregressive Models Enable Hierarchical Reinforcement Learning (2025)0.00
• Reusable Options Through Gradient-based Meta Learning (2022)0.00
• Discrete State-action Abstraction Via The Successor Representation (2022)0.00
• Learning Representations In Model-free Hierarchical Reinforcement Learning (2018)11.49
• Learning And Exploiting Multiple Subgoals For Fast Exploration In Hierarchical Reinforcement Learning (2019)0.00