Boosting Offline Reinforcement Learning With Residual Generative Modeling

Abstract

Offline reinforcement learning (RL) tries to learn the near-optimal policy with recorded offline experience without online exploration. Current offline RL research includes: 1) generative modeling, i.e., approximating a policy using fixed data; and 2) learning the state-action value function. While most research focuses on the state-action function part through reducing the bootstrapping error in value function approximation induced by the distribution shift of training data, the effects of error propagation in generative modeling have been neglected. In this paper, we analyze the error in generative modeling. We propose AQL (action-conditioned Q-learning), a residual generative model to reduce policy approximation error for offline RL. We show that our method can learn more accurate policy approximations in different benchmark datasets. In addition, we show that the proposed offline RL method can learn more competitive AI agents in complex control tasks under the multiplayer online ba

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywei2021boosting

Related papers

• ACL-QL: Adaptive Conservative Level In Q-learning For Offline Reinforcement Learning (2024)0.00
• Active Advantage-aligned Online Reinforcement Learning With Offline Data (2025)0.00
• Morel : Model-based Offline Reinforcement Learning (2020)0.00
• Expert-supervised Reinforcement Learning For Offline Policy Learning And Evaluation (2020)0.00
• Robust Offline Reinforcement Learning With Gradient Penalty And Constraint Relaxation (2022)0.00
• Hybrid RL: Using Both Offline And Online Data Can Make RL Efficient (2022)0.00
• An Optimistic Perspective On Offline Reinforcement Learning (2019)0.00
• Sample Efficient Active Algorithms For Offline Reinforcement Learning (2026)0.00