When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?

Abstract

Offline reinforcement learning (RL) algorithms can acquire effective policies by utilizing previously collected experience, without any online interaction. It is widely understood that offline RL is able to extract good policies even from highly suboptimal data, a scenario where imitation learning finds suboptimal solutions that do not improve over the demonstrator that generated the dataset. However, another common use case for practitioners is to learn from data that resembles demonstrations. In this case, one can choose to apply offline RL, but can also use behavioral cloning (BC) algorithms, which mimic a subset of the dataset via supervised learning. Therefore, it seems natural to ask: when can an offline RL method outperform BC with an equal amount of expert data, even when BC is a natural choice? To answer this question, we characterize the properties of environments that allow offline RL methods to perform better than BC methods, even when only provided with expert data. Additi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykumar2022when

Related papers

• Know Your Boundaries: The Necessity Of Explicit Behavioral Cloning In Offline RL (2022)0.00
• Reliable Conditioning Of Behavioral Cloning For Offline Reinforcement Learning (2022)0.00
• When Should We Prefer Decision Transformers For Offline Reinforcement Learning? (2023)0.00
• Expert Or Not? Assessing Data Quality In Offline Reinforcement Learning (2025)0.00
• Curriculum Offline Imitation Learning (2021)0.00
• Improving TD3-BC: Relaxed Policy Constraint For Offline Learning And Stable Online Fine-tuning (2022)0.00
• Behavior Prior Representation Learning For Offline Reinforcement Learning (2022)0.00
• Adaptive Behavior Cloning Regularization For Stable Offline-to-online Reinforcement Learning (2022)8.09