Improving TD3-BC: Relaxed Policy Constraint For Offline Learning And Stable Online Fine-tuning

Abstract

The ability to discover optimal behaviour from fixed data sets has the potential to transfer the successes of reinforcement learning (RL) to domains where data collection is acutely problematic. In this offline setting, a key challenge is overcoming overestimation bias for actions not present in data which, without the ability to correct for via interaction with the environment, can propagate and compound during training, leading to highly sub-optimal policies. One simple method to reduce this bias is to introduce a policy constraint via behavioural cloning (BC), which encourages agents to pick actions closer to the source data. By finding the right balance between RL and BC such approaches have been shown to be surprisingly effective while requiring minimal changes to the underlying algorithms they are based on. To date this balance has been held constant, but in this work we explore the idea of tipping this balance towards RL following initial training. Using TD3-BC, we demonstrate t

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybeeson2022improving

Related papers

• TD3 With Reverse KL Regularizer For Offline Reinforcement Learning From Mixed Datasets (2022)4.52
• B3C: A Minimalist Approach To Offline Multi-agent Reinforcement Learning (2025)0.00
• Reliable Conditioning Of Behavioral Cloning For Offline Reinforcement Learning (2022)0.00
• Adaptive Behavior Cloning Regularization For Stable Offline-to-online Reinforcement Learning (2022)8.09
• Know Your Boundaries: The Necessity Of Explicit Behavioral Cloning In Offline RL (2022)0.00
• When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? (2022)0.00
• Offline Retraining For Online RL: Decoupled Policy Learning To Mitigate Exploration Bias (2023)2.56
• Iteratively Refined Behavior Regularization For Offline Reinforcement Learning (2023)2.26