Off-policy RL Algorithms Can Be Sample-efficient For Continuous Control Via Sample Multiple Reuse

Abstract

Sample efficiency is one of the most critical issues for online reinforcement learning (RL). Existing methods achieve higher sample efficiency by adopting model-based methods, Q-ensemble, or better exploration mechanisms. We, instead, propose to train an off-policy RL agent via updating on a fixed sampled batch multiple times, thus reusing these samples and better exploiting them within a single optimization loop. We name our method sample multiple reuse (SMR). We theoretically show the properties of Q-learning with SMR, e.g., convergence. Furthermore, we incorporate SMR with off-the-shelf off-policy RL algorithms and conduct experiments on a variety of continuous control benchmarks. Empirical results show that SMR significantly boosts the sample efficiency of the base methods across most of the evaluated tasks without any hyperparameter tuning or additional tricks.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylyu2023off

Related papers

• Generalized Policy Improvement Algorithms With Theoretically Supported Sample Reuse (2022)5.24
• Robust On-policy Sampling For Data-efficient Policy Evaluation In Reinforcement Learning (2021)0.00
• Novelty-based Sample Reuse For Continuous Robotics Control (2024)0.00
• On Sample-efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, And Beyond (2024)0.00
• Learning Off-policy With Model-based Intrinsic Motivation For Active Online Exploration (2024)0.00
• Higher Replay Ratio Empowers Sample-efficient Multi-agent Reinforcement Learning (2024)0.00
• Enhancing Efficiency Of Safe Reinforcement Learning Via Sample Manipulation (2024)0.00
• Frugal Actor-critic: Sample Efficient Off-policy Deep Reinforcement Learning Using Unique Experiences (2024)0.00