Understanding When Dynamics-invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates

Abstract

Recently, data augmentation (DA) has emerged as a method for leveraging domain knowledge to inexpensively generate additional data in reinforcement learning (RL) tasks, often yielding substantial improvements in data efficiency. While prior work has demonstrated the utility of incorporating augmented data directly into model-free RL updates, it is not well-understood when a particular DA strategy will improve data efficiency. In this paper, we seek to identify general aspects of DA responsible for observed learning improvements. Our study focuses on sparse-reward tasks with dynamics-invariant data augmentation functions, serving as an initial step towards a more general understanding of DA and its integration into RL training. Experimentally, we isolate three relevant aspects of DA: state-action coverage, reward density, and the number of augmented transitions generated per update (the augmented replay ratio). From our experiments, we draw two conclusions: (1) increasing state-action c

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keycorrado2023understanding

Related papers

• A Recipe For Unbounded Data Augmentation In Visual Reinforcement Learning (2024)0.00
• DARA: Dynamics-aware Reward Augmentation In Offline Reinforcement Learning (2022)0.00
• Mocoda: Model-based Counterfactual Data Augmentation (2022)2.26
• Generalization Of Reinforcement Learning With Policy-aware Adversarial Data Augmentation (2021)0.00
• MAD-TD: Model-augmented Data Stabilizes High Update Ratio RL (2024)0.00
• Return Augmented Decision Transformer For Off-dynamics Reinforcement Learning (2024)0.00
• Equivariant Data Augmentation For Generalization In Offline Reinforcement Learning (2023)0.00
• Bootstrap Advantage Estimation For Policy Optimization In Reinforcement Learning (2022)0.00