Q-function Decomposition With Intervention Semantics With Factored Action Spaces

Abstract

Many practical reinforcement learning environments have a discrete factored action space that induces a large combinatorial set of actions, thereby posing significant challenges. Existing approaches leverage the regular structure of the action space and resort to a linear decomposition of Q-functions, which avoids enumerating all combinations of factored actions. In this paper, we consider Q-functions defined over a lower dimensional projected subspace of the original action space, and study the condition for the unbiasedness of decomposed Q-functions using causal effect estimation from the no unobserved confounder setting in causal statistics. This leads to a general scheme which we call action decomposed reinforcement learning that uses the projected Q-functions to approximate the Q-function in standard model-free reinforcement learning algorithms. The proposed approach is shown to improve sample complexity in a model-based reinforcement learning setting. We demonstrate improvements

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylee2025q

Related papers

• Leveraging Factored Action Spaces For Efficient Offline Reinforcement Learning In Healthcare (2023)2.26
• Reducing Action Space For Deep Reinforcement Learning Via Causal Effect Estimation (2025)0.00
• Achieving Sample And Computational Efficient Reinforcement Learning By Action Space Reduction Via Grouping (2023)0.00
• An Investigation Of Offline Reinforcement Learning In Factorisable Action Spaces (2024)0.00
• Adaptive Discretization For Episodic Reinforcement Learning In Metric Spaces (2019)2.26
• No Prior Mask: Eliminate Redundant Action For Deep Reinforcement Learning (2023)1.81
• Factorized Q-learning For Large-scale Multi-agent Systems (2018)11.58
• Concaveq: Non-monotonic Value Function Factorization Via Concave Representations In Deep Multi-agent Reinforcement Learning (2023)5.84