Importance Weighted Actor-critic For Optimal Conservative Offline Reinforcement Learning

Abstract

We propose A-Crab (Actor-Critic Regularized by Average Bellman error), a new practical algorithm for offline reinforcement learning (RL) in complex environments with insufficient data coverage. Our algorithm combines the marginalized importance sampling framework with the actor-critic paradigm, where the critic returns evaluations of the actor (policy) that are pessimistic relative to the offline data and have a small average (importance-weighted) Bellman error. Compared to existing methods, our algorithm simultaneously offers a number of advantages: (1) It achieves the optimal statistical rate of \(1/\sqrt\{N\}\) -- where \(N\) is the size of offline dataset -- in converging to the best policy covered in the offline dataset, even when combined with general function approximators. (2) It relies on a weaker average notion of policy coverage (compared to the \(\ell_\infty\) single-policy concentrability) that exploits the structure of policy visitations. (3) It outperforms the data-colle

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhu2023importance

Related papers

• Adversarially Trained Weighted Actor-critic For Safe Offline Reinforcement Learning (2024)0.00
• Provable Benefits Of Actor-critic Methods For Offline Reinforcement Learning (2021)0.00
• BRAC+: Improved Behavior Regularized Actor Critic For Offline Reinforcement Learning (2021)0.00
• Optimization Solution Functions As Deterministic Policies For Offline Reinforcement Learning (2024)0.00
• Offline-boosted Actor-critic: Adaptively Blending Optimal Historical Behaviors In Deep Off-policy RL (2024)0.00
• Optimal Conservative Offline RL With General Function Approximation Via Augmented Lagrangian (2022)0.00
• Relu To The Rescue: Improve Your On-policy Actor-critic With Positive Advantages (2023)0.00
• B3C: A Minimalist Approach To Offline Multi-agent Reinforcement Learning (2025)0.00