Exploration Implies Data Augmentation: Reachability And Generalisation In Contextual Mdps

Abstract

In the zero-shot policy transfer (ZSPT) setting for contextual Markov decision processes (MDP), agents train on a fixed set of contexts and must generalise to new ones. Recent work has argued and demonstrated that increased exploration can improve this generalisation, by training on more states in the training contexts. In this paper, we demonstrate that training on more states can indeed improve generalisation, but can come at a cost of reducing the accuracy of the learned value function which should not benefit generalisation. We hypothesise and demonstrate that using exploration to increase the agent's coverage while also increasing the accuracy improves generalisation even more. Inspired by this, we propose a method Explore-Go that implements an exploration phase at the beginning of each episode, which can be combined with existing on- and off-policy RL algorithms and significantly improves generalisation even in partially observable MDPs. We demonstrate the effectiveness of Explor

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyweltevrede2024exploration

Related papers

• Explore-go: Leveraging Exploration For Generalisation In Deep Reinforcement Learning (2024)0.00
• Investigating The Role Of Model-based Learning In Exploration And Transfer (2023)0.00
• Inferring Behavior-specific Context Improves Zero-shot Generalization In Reinforcement Learning (2024)0.95
• Active Exploration In Markov Decision Processes (2019)0.00
• Inverse Reinforcement Learning In Contextual Mdps (2019)8.82
• No-regret Exploration In Contextual Reinforcement Learning (2019)0.00
• Improved Bounds For Reward-agnostic And Reward-free Exploration (2026)0.00
• Experience Augmentation: Boosting And Accelerating Off-policy Multi-agent Reinforcement Learning (2020)0.00