Growing Action Spaces

Abstract

In complex tasks, such as those with large combinatorial action spaces, random exploration may be too inefficient to achieve meaningful learning progress. In this work, we use a curriculum of progressively growing action spaces to accelerate learning. We assume the environment is out of our control, but that the agent may set an internal curriculum by initially restricting its action space. Our approach uses off-policy reinforcement learning to estimate optimal value functions for multiple action spaces simultaneously and efficiently transfers data, value estimates, and state representations from restricted action spaces to the full task. We show the efficacy of our approach in proof-of-concept control tasks and on challenging large-scale StarCraft micromanagement tasks with large, multi-agent action spaces.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfarquhar2019growing

Related papers

• Episodic Exploration For Deep Deterministic Policies: An Application To Starcraft Micromanagement Tasks (2016)0.00
• Starcraft Micromanagement With Reinforcement Learning And Curriculum Transfer Learning (2018)16.19
• Learning In Complex Action Spaces Without Policy Gradients (2024)0.00
• Open-ended Learning Leads To Generally Capable Agents (2021)0.00
• Action-adaptive Continual Learning: Enabling Policy Generalization Under Dynamic Action Spaces (2025)0.00
• Marginal Policy Gradients: A Unified Family Of Estimators For Bounded Action Spaces With Applications (2018)0.00
• Extremum-seeking Action Selection For Accelerating Policy Optimization (2024)0.00
• Reducing Action Space For Deep Reinforcement Learning Via Causal Effect Estimation (2025)0.00