Skill-critic: Refining Learned Skills For Hierarchical Reinforcement Learning

Abstract

Hierarchical reinforcement learning (RL) can accelerate long-horizon decision-making by temporally abstracting a policy into multiple levels. Promising results in sparse reward environments have been seen with skills, i.e. sequences of primitive actions. Typically, a skill latent space and policy are discovered from offline data. However, the resulting low-level policy can be unreliable due to low-coverage demonstrations or distribution shifts. As a solution, we propose the Skill-Critic algorithm to fine-tune the low-level policy in conjunction with high-level skill selection. Our Skill-Critic algorithm optimizes both the low-level and high-level policies; these policies are initialized and regularized by the latent space learned from offline demonstrations to guide the parallel policy optimization. We validate Skill-Critic in multiple sparse-reward RL environments, including a new sparse-reward autonomous racing task in Gran Turismo Sport. The experiments show that Skill-Critic's low-

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keyhao2023skill

Related papers

• Hypothesis-driven Skill Discovery For Hierarchical Deep Reinforcement Learning (2019)2.26
• Hierarchical And Interpretable Skill Acquisition In Multi-task Reinforcement Learning (2017)0.00
• Hierarchical Reinforcement Learning With Advantage-based Auxiliary Rewards (2019)0.00
• Reinforcement Learning From Hierarchical Critics (2019)8.09
• When Do Skills Help Reinforcement Learning? A Theoretical Analysis Of Temporal Abstractions (2024)0.00
• Self-improving Skill Learning For Robust Skill-based Meta-reinforcement Learning (2025)0.00
• Skills: Adaptive Skill Sequencing For Efficient Temporally-extended Exploration (2022)0.00
• Developing Cooperative Policies For Multi-stage Reinforcement Learning Tasks (2022)0.00