Statistically Efficient Advantage Learning For Offline Reinforcement Learning In Infinite Horizons

Abstract

We consider reinforcement learning (RL) methods in offline domains without additional online data collection, such as mobile health applications. Most of existing policy optimization algorithms in the computer science literature are developed in online settings where data are easy to collect or simulate. Their generalizations to mobile health applications with a pre-collected offline dataset remain unknown. The aim of this paper is to develop a novel advantage learning framework in order to efficiently use pre-collected data for policy optimization. The proposed method takes an optimal Q-estimator computed by any existing state-of-the-art RL algorithms as input, and outputs a new policy whose value is guaranteed to converge at a faster rate than the policy derived based on the initial Q-estimator. Extensive numerical experiments are conducted to back up our theoretical findings. A Python implementation of our proposed method is available at https://github.com/leyuanheart/SEAL.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyshi2022statistically

Code

• leyuanheart/SEAL—

Related papers

• Active Advantage-aligned Online Reinforcement Learning With Offline Data (2025)0.00
• Expert-supervised Reinforcement Learning For Offline Policy Learning And Evaluation (2020)0.00
• Deployment-efficient Reinforcement Learning Via Model-based Offline Optimization (2020)0.00
• AWAC: Accelerating Online Reinforcement Learning With Offline Datasets (2020)0.00
• A2PO: Towards Effective Offline Reinforcement Learning From An Advantage-aware Perspective (2024)1.69
• Offline Policy Evaluation For Reinforcement Learning With Adaptively Collected Data (2023)0.00
• Hybrid RL: Using Both Offline And Online Data Can Make RL Efficient (2022)0.00
• Reinforcement Learning For Individual Optimal Policy From Heterogeneous Data (2025)0.00