Leveraging Factored Action Spaces For Efficient Offline Reinforcement Learning In Healthcare

Abstract

Many reinforcement learning (RL) applications have combinatorial action spaces, where each action is a composition of sub-actions. A standard RL approach ignores this inherent factorization structure, resulting in a potential failure to make meaningful inferences about rarely observed sub-action combinations; this is particularly problematic for offline settings, where data may be limited. In this work, we propose a form of linear Q-function decomposition induced by factored action spaces. We study the theoretical properties of our approach, identifying scenarios where it is guaranteed to lead to zero bias when used to approximate the Q-function. Outside the regimes with theoretical guarantees, we show that our approach can still be useful because it leads to better sample efficiency without necessarily sacrificing policy optimality, allowing us to achieve a better bias-variance trade-off. Across several offline RL problems using simulators and real-world datasets motivated by healthca

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keytang2023leveraging

Related papers

• An Investigation Of Offline Reinforcement Learning In Factorisable Action Spaces (2024)0.00
• Q-function Decomposition With Intervention Semantics With Factored Action Spaces (2025)0.00
• FAST-Q: Fast-track Exploration With Adversarially Balanced State Representations For Counterfactual Action Estimation In Offline Reinforcement Learning (2025)0.00
• Statistically Efficient Advantage Learning For Offline Reinforcement Learning In Infinite Horizons (2022)0.00
• Goal-conditioned Offline Reinforcement Learning Through State Space Partitioning (2023)2.26
• Federated Offline Reinforcement Learning (2022)0.00
• Boosting Offline Reinforcement Learning With Residual Generative Modeling (2021)0.00
• An Empirical Study Of Representation Learning For Reinforcement Learning In Healthcare (2020)0.00