Coptidice: Offline Constrained Reinforcement Learning Via Stationary Distribution Correction Estimation

Abstract

We consider the offline constrained reinforcement learning (RL) problem, in which the agent aims to compute a policy that maximizes expected return while satisfying given cost constraints, learning only from a pre-collected dataset. This problem setting is appealing in many real-world scenarios, where direct interaction with the environment is costly or risky, and where the resulting policy should comply with safety constraints. However, it is challenging to compute a policy that guarantees satisfying the cost constraints in the offline RL setting, since the off-policy evaluation inherently has an estimation error. In this paper, we present an offline constrained RL algorithm that optimizes the policy in the space of the stationary distribution. Our algorithm, COptiDICE, directly estimates the stationary distribution corrections of the optimal policy with respect to returns, while constraining the cost upper bound, with the goal of yielding a cost-conservative policy for actual constra

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylee2022coptidice

Related papers

• Optidice: Offline Policy Optimization Via Stationary Distribution Correction Estimation (2021)0.00
• Learning From Sparse Offline Datasets Via Conservative Density Estimation (2024)0.00
• Regularizing A Model-based Policy Stationary Distribution To Stabilize Offline Reinforcement Learning (2022)0.00
• Alberdice: Addressing Out-of-distribution Joint Actions In Offline Multi-agent RL Via Alternating Stationary Distribution Correction Estimation (2023)0.00
• State-constrained Offline Reinforcement Learning (2024)0.00
• Dualdice: Behavior-agnostic Estimation Of Discounted Stationary Distribution Corrections (2019)0.00
• Lobsdice: Offline Learning From Observation Via Stationary Distribution Correction Estimation (2022)0.00
• Comadice: Offline Cooperative Multi-agent Reinforcement Learning With Stationary Distribution Shift Regularization (2024)0.00