Guided Online Distillation: Promoting Safe Reinforcement Learning By Offline Demonstration

Abstract

Safe Reinforcement Learning (RL) aims to find a policy that achieves high rewards while satisfying cost constraints. When learning from scratch, safe RL agents tend to be overly conservative, which impedes exploration and restrains the overall performance. In many realistic tasks, e.g. autonomous driving, large-scale expert demonstration data are available. We argue that extracting expert policy from offline data to guide online exploration is a promising solution to mitigate the conserveness issue. Large-capacity models, e.g. decision transformers (DT), have been proven to be competent in offline policy learning. However, data collected in real-world scenarios rarely contain dangerous cases (e.g., collisions), which makes it prohibitive for the policies to learn safety concepts. Besides, these bulk policy networks cannot meet the computation speed requirements at inference time on real-world tasks such as autonomous driving. To this end, we propose Guided Online Distillation (GOLD), a

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keyli2023guided

Related papers

• Towards Fast Safe Online Reinforcement Learning Via Policy Finetuning (2024)0.00
• AWAC: Accelerating Online Reinforcement Learning With Offline Datasets (2020)0.00
• Hundreds Guide Millions: Adaptive Offline Reinforcement Learning With Expert Guidance (2023)7.50
• Towards Data-driven Offline Simulations For Online Reinforcement Learning (2022)0.00
• Reinforcement Learning With Sparse Rewards Using Guidance From Offline Demonstration (2022)0.00
• DEFENDER: Dtw-based Episode Filtering Using Demonstrations For Enhancing RL Safety (2023)0.00
• Reinforcement Learning By Guided Safe Exploration (2023)5.24
• Offline Retraining For Online RL: Decoupled Policy Learning To Mitigate Exploration Bias (2023)2.56