Know Your Boundaries: The Necessity Of Explicit Behavioral Cloning In Offline RL

Abstract

We introduce an offline reinforcement learning (RL) algorithm that explicitly clones a behavior policy to constrain value learning. In offline RL, it is often important to prevent a policy from selecting unobserved actions, since the consequence of these actions cannot be presumed without additional information about the environment. One straightforward way to implement such a constraint is to explicitly model a given data distribution via behavior cloning and directly force a policy not to select uncertain actions. However, many offline RL methods instantiate the constraint indirectly -- for example, pessimistic value estimation -- due to a concern about errors when modeling a potentially complex behavior policy. In this work, we argue that it is not only viable but beneficial to explicitly model the behavior policy for offline RL because the constraint can be realized in a stable way with the trained model. We first suggest a theoretical framework that allows us to incorporate behavi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygoo2022know

Related papers

• Adaptive Behavior Cloning Regularization For Stable Offline-to-online Reinforcement Learning (2022)8.09
• When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? (2022)0.00
• Reliable Conditioning Of Behavioral Cloning For Offline Reinforcement Learning (2022)0.00
• Behavior Prior Representation Learning For Offline Reinforcement Learning (2022)0.00
• Improving TD3-BC: Relaxed Policy Constraint For Offline Learning And Stable Online Fine-tuning (2022)0.00
• Offline Behavioral Data Selection (2025)0.00
• Curriculum Offline Imitation Learning (2021)0.00
• B3C: A Minimalist Approach To Offline Multi-agent Reinforcement Learning (2025)0.00