Hybrid RL: Using Both Offline And Online Data Can Make RL Efficient

Abstract

We consider a hybrid reinforcement learning setting (Hybrid RL), in which an agent has access to an offline dataset and the ability to collect experience via real-world online interaction. The framework mitigates the challenges that arise in both pure offline and online RL settings, allowing for the design of simple and highly effective algorithms, in both theory and practice. We demonstrate these advantages by adapting the classical Q learning/iteration algorithm to the hybrid setting, which we call Hybrid Q-Learning or Hy-Q. In our theoretical results, we prove that the algorithm is both computationally and statistically efficient whenever the offline dataset supports a high-quality policy and the environment has bounded bilinear rank. Notably, we require no assumptions on the coverage provided by the initial distribution, in contrast with guarantees for policy gradient/iteration methods. In our experimental results, we show that Hy-Q with neural network function approximation outper

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysong2022hybrid

Related papers

• Reward-agnostic Fine-tuning: Provable Statistical Benefits Of Hybrid Reinforcement Learning (2023)0.00
• When To Trust Your Simulator: Dynamics-aware Hybrid Offline-and-online Reinforcement Learning (2022)2.26
• Boosting Offline Reinforcement Learning With Residual Generative Modeling (2021)0.00
• Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency From Shifted-dynamics Data (2024)0.00
• H2O+: An Improved Framework For Hybrid Offline-and-online RL With Dynamics Gaps (2023)0.00
• Statistically Efficient Advantage Learning For Offline Reinforcement Learning In Infinite Horizons (2022)0.00
• Cal-ql: Calibrated Offline RL Pre-training For Efficient Online Fine-tuning (2023)2.26
• Constraints Penalized Q-learning For Safe Offline Reinforcement Learning (2021)0.00