Beyond OOD State Actions: Supported Cross-domain Offline Reinforcement Learning

Abstract

Offline reinforcement learning (RL) aims to learn a policy using only pre-collected and fixed data. Although avoiding the time-consuming online interactions in RL, it poses challenges for out-of-distribution (OOD) state actions and often suffers from data inefficiency for training. Despite many efforts being devoted to addressing OOD state actions, the latter (data inefficiency) receives little attention in offline RL. To address this, this paper proposes the cross-domain offline RL, which assumes offline data incorporate additional source-domain data from varying transition dynamics (environments), and expects it to contribute to the offline data efficiency. To do so, we identify a new challenge of OOD transition dynamics, beyond the common OOD state actions issue, when utilizing cross-domain offline data. Then, we propose our method BOSA, which employs two support-constrained objectives to address the above OOD issues. Through extensive experiments in the cross-domain offline RL sett

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2023beyond

Related papers

• Offline Reinforcement Learning With OOD State Correction And OOD Action Suppression (2024)0.00
• Planning To Go Out-of-distribution In Offline-to-online Reinforcement Learning (2023)0.00
• Towards Data-driven Offline Simulations For Online Reinforcement Learning (2022)0.00
• State-constrained Offline Reinforcement Learning (2024)0.00
• BRAC+: Improved Behavior Regularized Actor Critic For Offline Reinforcement Learning (2021)0.00
• AWAC: Accelerating Online Reinforcement Learning With Offline Datasets (2020)0.00
• FOCAL: Efficient Fully-offline Meta-reinforcement Learning Via Distance Metric Learning And Behavior Regularization (2020)0.00
• Dual-robust Cross-domain Offline Reinforcement Learning Against Dynamics Shifts (2025)0.00