Hundreds Guide Millions: Adaptive Offline Reinforcement Learning With Expert Guidance

Abstract

Offline reinforcement learning (RL) optimizes the policy on a previously collected dataset without any interactions with the environment, yet usually suffers from the distributional shift problem. To mitigate this issue, a typical solution is to impose a policy constraint on a policy improvement objective. However, existing methods generally adopt a ``one-size-fits-all'' practice, i.e., keeping only a single improvement-constraint balance for all the samples in a mini-batch or even the entire offline dataset. In this work, we argue that different samples should be treated with different policy constraint intensities. Based on this idea, a novel plug-in approach named Guided Offline RL (GORL) is proposed. GORL employs a guiding network, along with only a few expert demonstrations, to adaptively determine the relative importance of the policy improvement and policy constraint for every sample. We theoretically prove that the guidance provided by our method is rational and near-optimal. E

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keyyang2023hundreds

Related papers

• Guided Online Distillation: Promoting Safe Reinforcement Learning By Offline Demonstration (2023)4.52
• Expert-supervised Reinforcement Learning For Offline Policy Learning And Evaluation (2020)0.00
• A Policy-guided Imitation Approach For Offline Reinforcement Learning (2022)0.00
• Reinforcement Learning With Sparse Rewards Using Guidance From Offline Demonstration (2022)0.00
• Robust Offline Reinforcement Learning With Gradient Penalty And Constraint Relaxation (2022)0.00
• Optimality Inductive Biases And Agnostic Guidelines For Offline Reinforcement Learning (2021)0.00
• Morel : Model-based Offline Reinforcement Learning (2020)0.00
• Enhancing Online Reinforcement Learning With Meta-learned Objective From Offline Data (2025)0.00