Reducing Action Space For Deep Reinforcement Learning Via Causal Effect Estimation

Abstract

Intelligent decision-making within large and redundant action spaces remains challenging in deep reinforcement learning. Considering similar but ineffective actions at each step can lead to repetitive and unproductive trials. Existing methods attempt to improve agent exploration by reducing or penalizing redundant actions, yet they fail to provide quantitative and reliable evidence to determine redundancy. In this paper, we propose a method to improve exploration efficiency by estimating the causal effects of actions. Unlike prior methods, our approach offers quantitative results regarding the causality of actions for one-step transitions. We first pre-train an inverse dynamics model to serve as prior knowledge of the environment. Subsequently, we classify actions across the entire action space at each time step and estimate the causal effect of each action to suppress redundant actions during exploration. We provide a theoretical analysis to demonstrate the effectiveness of our method

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2025reducing

Related papers

• No Prior Mask: Eliminate Redundant Action For Deep Reinforcement Learning (2023)1.81
• Disentangling Causal Effects For Hierarchical Reinforcement Learning (2020)0.00
• Goal Discovery With Causal Capacity For Efficient Reinforcement Learning (2025)0.00
• Causal Influence Detection For Improving Efficiency In Reinforcement Learning (2021)0.00
• Achieving Sample And Computational Efficient Reinforcement Learning By Action Space Reduction Via Grouping (2023)0.00
• Boosting Efficiency In Task-agnostic Exploration Through Causal Knowledge (2024)0.00
• Q-function Decomposition With Intervention Semantics With Factored Action Spaces (2025)0.00
• Data-driven Evaluation Of Training Action Space For Reinforcement Learning (2022)0.00