Learning Safe Policies With Expert Guidance

Abstract

We propose a framework for ensuring safe behavior of a reinforcement learning agent when the reward function may be difficult to specify. In order to do this, we rely on the existence of demonstrations from expert policies, and we provide a theoretical framework for the agent to optimize in the space of rewards consistent with its existing knowledge. We propose two methods to solve the resulting optimization: an exact ellipsoid-based method and a method in the spirit of the "follow-the-perturbed-leader" algorithm. Our experiments demonstrate the behavior of our algorithm in both discrete and continuous problems. The trained agent safely avoids states with potential negative effects while imitating the behavior of the expert in the other states.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhuang2018learning

Related papers

• Reward-conditioned Policies (2019)0.00
• Safe Driving Via Expert Guided Policy Optimization (2021)0.00
• Reinforcement Learning By Guided Safe Exploration (2023)5.24
• Policy Gradient From Demonstration And Curiosity (2020)0.00
• Conservative Exploration For Policy Optimization Via Off-policy Policy Evaluation (2023)0.00
• Probabilistic Counterexample Guidance For Safer Reinforcement Learning (extended Version) (2023)0.00
• Safety Correction From Baseline: Towards The Risk-aware Policy In Robotics Via Dual-agent Reinforcement Learning (2022)3.58
• Joint Learning Of Policy With Unknown Temporal Constraints For Safe Reinforcement Learning (2023)0.00