A2PO: Towards Effective Offline Reinforcement Learning From An Advantage-aware Perspective

Abstract

Offline reinforcement learning endeavors to leverage offline datasets to craft effective agent policy without online interaction, which imposes proper conservative constraints with the support of behavior policies to tackle the out-of-distribution problem. However, existing works often suffer from the constraint conflict issue when offline datasets are collected from multiple behavior policies, i.e., different behavior policies may exhibit inconsistent actions with distinct returns across the state space. To remedy this issue, recent advantage-weighted methods prioritize samples with high advantage values for agent training while inevitably ignoring the diversity of behavior policy. In this paper, we introduce a novel Advantage-Aware Policy Optimization (A2PO) method to explicitly construct advantage-aware policy constraints for offline learning under mixed-quality datasets. Specifically, A2PO employs a conditional variational auto-encoder to disentangle the action distributions of int

Stats

• citations0
• S2 citations—
• github stars6
• HF likes0
• heat score1.69
• arxiv keyqing2024a2po

Related papers

• Adaptive Advantage-guided Policy Regularization For Offline Reinforcement Learning (2024)3.09
• Active Advantage-aligned Online Reinforcement Learning With Offline Data (2025)0.00
• Constrained Latent Action Policies For Model-based Offline Reinforcement Learning (2024)0.00
• POPO: Pessimistic Offline Policy Optimization (2020)5.24
• AWAC: Accelerating Online Reinforcement Learning With Offline Datasets (2020)0.00
• A Behavior Regularized Implicit Policy For Offline Reinforcement Learning (2022)0.00
• Statistically Efficient Advantage Learning For Offline Reinforcement Learning In Infinite Horizons (2022)0.00
• Optimality Inductive Biases And Agnostic Guidelines For Offline Reinforcement Learning (2021)0.00