State-constrained Offline Reinforcement Learning

Abstract

Traditional offline reinforcement learning (RL) methods predominantly operate in a batch-constrained setting. This confines the algorithms to a specific state-action distribution present in the dataset, reducing the effects of distributional shift but restricting the policy to seen actions. In this paper, we alleviate this limitation by introducing state-constrained offline RL, a novel framework that focuses solely on the dataset's state distribution. This approach allows the policy to take high-quality out-of-distribution actions that lead to in-distribution states, significantly enhancing learning potential. The proposed setting not only broadens the learning horizon but also improves the ability to combine different trajectories from the dataset effectively, a desirable property inherent in offline RL. Our research is underpinned by theoretical findings that pave the way for subsequent advancements in this area. Additionally, we introduce StaCQ, a deep learning algorithm that achiev

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhepburn2024state

Related papers

• Constraints Penalized Q-learning For Safe Offline Reinforcement Learning (2021)0.00
• Goal-conditioned Offline Reinforcement Learning Through State Space Partitioning (2023)2.26
• Regularizing A Model-based Policy Stationary Distribution To Stabilize Offline Reinforcement Learning (2022)0.00
• Constrained Latent Action Policies For Model-based Offline Reinforcement Learning (2024)0.00
• Robust Offline Reinforcement Learning With Gradient Penalty And Constraint Relaxation (2022)0.00
• Policy Constraint By Only Support Constraint For Offline Reinforcement Learning (2025)0.00
• Bridging Distributionally Robust Learning And Offline RL: An Approach To Mitigate Distribution Shift And Partial Data Coverage (2023)0.00
• Believe What You See: Implicit Constraint Approach For Offline Multi-agent Reinforcement Learning (2021)0.00