Neighborhood Mixup Experience Replay: Local Convex Interpolation For Improved Sample Efficiency In Continuous Control Tasks

Abstract

Experience replay plays a crucial role in improving the sample efficiency of deep reinforcement learning agents. Recent advances in experience replay propose using Mixup (Zhang et al., 2018) to further improve sample efficiency via synthetic sample generation. We build upon this technique with Neighborhood Mixup Experience Replay (NMER), a geometrically-grounded replay buffer that interpolates transitions with their closest neighbors in state-action space. NMER preserves a locally linear approximation of the transition manifold by only applying Mixup between transitions with vicinal state-action features. Under NMER, a given transition's set of state action neighbors is dynamic and episode agnostic, in turn encouraging greater policy generalizability via inter-episode interpolation. We combine our approach with recent off-policy deep reinforcement learning algorithms and evaluate on continuous control environments. We observe that NMER improves sample efficiency by an average 94% (TD3)

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysander2022neighborhood

Related papers

• CUER: Corrected Uniform Experience Replay For Off-policy Continuous Deep Reinforcement Learning Algorithms (2024)0.00
• Accmer: Accelerating Multi-agent Experience Replay With Cache Locality-aware Prioritization (2023)5.24
• Introspective Experience Replay: Look Back When Surprised (2022)0.00
• Experience Replay Using Transition Sequences (2017)8.82
• Learning To Sample With Local And Global Contexts In Experience Replay Buffer (2020)0.00
• On The Convergence Of Experience Replay In Policy Optimization: Characterizing Bias, Variance, And Finite-time Convergence (2021)0.00
• Replay For Safety (2021)0.00
• Importance Mixing: Improving Sample Reuse In Evolutionary Policy Search Methods (2018)0.00