Option Discovery In Hierarchical Reinforcement Learning Using Spatio-temporal Clustering

Abstract

This paper introduces an automated skill acquisition framework in reinforcement learning which involves identifying a hierarchical description of the given task in terms of abstract states and extended actions between abstract states. Identifying such structures present in the task provides ways to simplify and speed up reinforcement learning algorithms. These structures also help to generalize such algorithms over multiple tasks without relearning policies from scratch. We use ideas from dynamical systems to find metastable regions in the state space and associate them with abstract states. The spectral clustering algorithm PCCA+ is used to identify suitable abstractions aligned to the underlying structure. Skills are defined in terms of the sequence of actions that lead to transitions between such abstract states. The connectivity information from PCCA+ is used to generate these skills or options. These skills are independent of the learning task and can be efficiently reused across

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysrinivas2016option

Related papers

• Disentangled Unsupervised Skill Discovery For Efficient Hierarchical Reinforcement Learning (2024)0.00
• Hypothesis-driven Skill Discovery For Hierarchical Deep Reinforcement Learning (2019)2.26
• Hierarchical And Interpretable Skill Acquisition In Multi-task Reinforcement Learning (2017)0.00
• Hierarchical Cooperative Multi-agent Reinforcement Learning With Skill Discovery (2019)5.24
• Hyperbolic Embeddings For Learning Options In Hierarchical Reinforcement Learning (2018)0.00
• Learning Representations In Model-free Hierarchical Reinforcement Learning (2018)11.49
• When Do Skills Help Reinforcement Learning? A Theoretical Analysis Of Temporal Abstractions (2024)0.00
• Skill-critic: Refining Learned Skills For Hierarchical Reinforcement Learning (2023)7.50