MAD-TD: Model-augmented Data Stabilizes High Update Ratio RL

Abstract

Building deep reinforcement learning (RL) agents that find a good policy with few samples has proven notoriously challenging. To achieve sample efficiency, recent work has explored updating neural networks with large numbers of gradient steps for every new sample. While such high update-to-data (UTD) ratios have shown strong empirical performance, they also introduce instability to the training process. Previous approaches need to rely on periodic neural network parameter resets to address this instability, but restarting the training process is infeasible in many real-world applications and requires tuning the resetting interval. In this paper, we focus on one of the core difficulties of stable training with limited samples: the inability of learned value functions to generalize to unobserved on-policy actions. We mitigate this issue directly by augmenting the off-policy RL training process with a small amount of data generated from a learned world model. Our method, Model-Augmented D

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvoelcker2024mad

Related papers

• Dynamic Update-to-data Ratio: Minimizing World Model Overfitting (2023)0.00
• Model-based Offline Reinforcement Learning With Adversarial Data Augmentation (2025)0.00
• Moments Matter:stabilizing Policy Optimization Using Return Distributions (2026)0.00
• Dissecting Deep RL With High Update Ratios: Combatting Value Divergence (2024)0.00
• Regularizing A Model-based Policy Stationary Distribution To Stabilize Offline Reinforcement Learning (2022)0.00
• Experience Augmentation: Boosting And Accelerating Off-policy Multi-agent Reinforcement Learning (2020)0.00
• Understanding When Dynamics-invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates (2023)0.00
• Live In The Moment: Learning Dynamics Model Adapted To Evolving Policy (2022)0.00