Exploring With Sticky Mittens: Reinforcement Learning With Expert Interventions Via Option Templates

Abstract

Long horizon robot learning tasks with sparse rewards pose a significant challenge for current reinforcement learning algorithms. A key feature enabling humans to learn challenging control tasks is that they often receive expert intervention that enables them to understand the high-level structure of the task before mastering low-level control actions. We propose a framework for leveraging expert intervention to solve long-horizon reinforcement learning tasks. We consider *option templates*, which are specifications encoding a potential option that can be trained using reinforcement learning. We formulate expert intervention as allowing the agent to execute option templates before learning an implementation. This enables them to use an option, before committing costly resources to learning it. We evaluate our approach on three challenging reinforcement learning problems, showing that it outperforms state-of-the-art approaches by two orders of magnitude. Videos of trained agents and our

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydutta2022exploring

Related papers

• LESSON: Learning To Integrate Exploration Strategies For Reinforcement Learning Via An Option Framework (2023)0.00
• Optimal Options For Multi-task Reinforcement Learning Under Time Constraints (2020)0.00
• A Provably Efficient Option-based Algorithm For Both High-level And Low-level Learning (2024)0.00
• A Hierarchical Reinforcement Learning Method For Persistent Time-sensitive Tasks (2016)0.00
• Autonomous Option Invention For Continual Hierarchical Reinforcement Learning And Planning (2024)2.26
• Online Matching Via Reinforcement Learning: An Expert Policy Orchestration Strategy (2025)0.00
• First-explore, Then Exploit: Meta-learning To Solve Hard Exploration-exploitation Trade-offs (2023)0.00
• Learn The Ropes, Then Trust The Wins: Self-imitation With Progressive Exploration For Agentic Reinforcement Learning (2025)0.00