Is Exploration Or Optimization The Problem For Deep Reinforcement Learning?

Abstract

In the era of deep reinforcement learning, making progress is more complex, as the collected experience must be compressed into a deep model for future exploitation and sampling. Many papers have shown that training a deep learning policy under the changing state and action distribution leads to sub-optimal performance, or even collapse. This naturally leads to the concern that even if the community creates improved exploration algorithms or reward objectives, will those improvements fall on the \textit\{deaf ears\} of optimization difficulties. This work proposes a new \textit\{practical\} sub-optimality estimator to determine optimization limitations of deep reinforcement learning algorithms. Through experiments across environments and RL algorithms, it is shown that the difference between the best experience generated is 2-3\(\times\) better than the policies' learned performance. This large difference indicates that deep RL methods only exploit half of the good experience they gene

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyberseth2025is

Related papers

• Exploration Conscious Reinforcement Learning Revisited (2018)0.00
• Satisficing Exploration For Deep Reinforcement Learning (2024)0.00
• Is Exploration All You Need? Effective Exploration Characteristics For Transfer In Reinforcement Learning (2024)0.00
• Conservative Exploration For Policy Optimization Via Off-policy Policy Evaluation (2023)0.00
• Exploration In Feature Space For Reinforcement Learning (2017)0.00
• Decoupled Exploration And Exploitation Policies For Sample-efficient Reinforcement Learning (2021)0.00
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00
• Minimax-optimal Reward-agnostic Exploration In Reinforcement Learning (2023)0.00