Adaptive Advantage-guided Policy Regularization For Offline Reinforcement Learning

Abstract

In offline reinforcement learning, the challenge of out-of-distribution (OOD) is pronounced. To address this, existing methods often constrain the learned policy through policy regularization. However, these methods often suffer from the issue of unnecessary conservativeness, hampering policy improvement. This occurs due to the indiscriminate use of all actions from the behavior policy that generates the offline dataset as constraints. The problem becomes particularly noticeable when the quality of the dataset is suboptimal. Thus, we propose Adaptive Advantage-guided Policy Regularization (A2PR), obtaining high-advantage actions from an augmented behavior policy combined with VAE to guide the learned policy. A2PR can select high-advantage actions that differ from those present in the dataset, while still effectively maintaining conservatism from OOD actions. This is achieved by harnessing the VAE capacity to generate samples matching the distribution of the data points. We theoreticall

Stats

• citations0
• S2 citations—
• github stars34
• HF likes0
• heat score3.09
• arxiv keyliu2024adaptive

Related papers

• A2PO: Towards Effective Offline Reinforcement Learning From An Advantage-aware Perspective (2024)1.69
• Policy Regularization With Dataset Constraint For Offline Reinforcement Learning (2023)0.00
• A Behavior Regularized Implicit Policy For Offline Reinforcement Learning (2022)0.00
• Active Advantage-aligned Online Reinforcement Learning With Offline Data (2025)0.00
• Iteratively Refined Behavior Regularization For Offline Reinforcement Learning (2023)2.26
• Offline Policy Optimization In RL With Variance Regularizaton (2022)0.00
• Constrained Latent Action Policies For Model-based Offline Reinforcement Learning (2024)0.00
• Regularizing A Model-based Policy Stationary Distribution To Stabilize Offline Reinforcement Learning (2022)0.00