Directed Policy Gradient For Safe Reinforcement Learning With Human Advice

Abstract

Many currently deployed Reinforcement Learning agents work in an environment shared with humans, be them co-workers, users or clients. It is desirable that these agents adjust to people's preferences, learn faster thanks to their help, and act safely around them. We argue that most current approaches that learn from human feedback are unsafe: rewarding or punishing the agent a-posteriori cannot immediately prevent it from wrong-doing. In this paper, we extend Policy Gradient to make it robust to external directives, that would otherwise break the fundamentally on-policy nature of Policy Gradient. Our technique, Directed Policy Gradient (DPG), allows a teacher or backup policy to override the agent before it acts undesirably, while allowing the agent to leverage human advice or directives to learn faster. Our experiments demonstrate that DPG makes the agent learn much faster than reward-based approaches, while requiring an order of magnitude less advice.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyplisnier2018directed

Related papers

• Smoothing Policies And Safe Policy Gradients (2019)7.50
• Influencing Reinforcement Learning Through Natural Language Guidance (2021)0.00
• Pref-guide: Continual Policy Learning From Real-time Human Feedback Via Preference-based Learning (2025)0.00
• Direct Policy Gradients: Direct Optimization Of Policies In Discrete Action Spaces (2019)0.00
• PC-PG: Policy Cover Directed Exploration For Provable Policy Gradient Learning (2020)0.00
• Deep Reinforcement Learning With Feedback-based Exploration (2019)5.84
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00