Novelty-based Sample Reuse For Continuous Robotics Control
2024 · Ke Duan, Kai Yang, Houde Liu, et al.
Abstract
In reinforcement learning, agents collect state information and rewards through environmental interactions, essential for policy refinement. This process is notably time-consuming, especially in complex robotic simulations and real-world applications. Traditional algorithms usually re-engage with the environment after processing a single batch of samples, thereby failing to fully capitalize on historical data. However, frequently observed states, with reliable value estimates, require minimal updates; in contrast, rare observed states necessitate more intensive updates for achieving accurate value estimations. To address uneven sample utilization, we propose Novelty-guided Sample Reuse (NSR). NSR provides extra updates for infrequent, novel states and skips additional updates for frequent states, maximizing sample use before interacting with the environment again. Our experiments show that NSR improves the convergence rate and success rate of algorithms without significantly increasing
Authors
(none)
Tags
Stats
Related papers
- Novelty-guided Data Reuse For Efficient And Diversified Multi-agent Reinforcement Learning (2024)2.26
- Off-policy RL Algorithms Can Be Sample-efficient For Continuous Control Via Sample Multiple Reuse (2023)0.00
- PNS: Population-guided Novelty Search For Reinforcement Learning In Hard Exploration Environments (2018)7.16
- Generalized Policy Improvement Algorithms With Theoretically Supported Sample Reuse (2022)5.24
- Frugal Actor-critic: Sample Efficient Off-policy Deep Reinforcement Learning Using Unique Experiences (2024)0.00
- Learning Off-policy With Model-based Intrinsic Motivation For Active Online Exploration (2024)0.00
- Learning Robust And Adaptive Real-world Continuous Control Using Simulation And Transfer Learning (2018)0.00
- Novelty Search For Deep Reinforcement Learning Policy Network Weights By Action Sequence Edit Metric Distance (2019)8.09