Some Supervision Required: Incorporating Oracle Policies In Reinforcement Learning Via Epistemic Uncertainty Metrics

Abstract

An inherent problem of reinforcement learning is performing exploration of an environment through random actions, of which a large portion can be unproductive. Instead, exploration can be improved by initializing the learning policy with an existing (previously learned or hard-coded) oracle policy, offline data, or demonstrations. In the case of using an oracle policy, it can be unclear how best to incorporate the oracle policy's experience into the learning policy in a way that maximizes learning sample efficiency. In this paper, we propose a method termed Critic Confidence Guided Exploration (CCGE) for incorporating such an oracle policy into standard actor-critic reinforcement learning algorithms. More specifically, CCGE takes in the oracle policy's actions as suggestions and incorporates this information into the learning scheme when uncertainty is high, while ignoring it when the uncertainty is low. CCGE is agnostic to methods of estimating uncertainty, and we show that it is equa

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytai2022some

Related papers

• Think Outside The Policy: In-context Steered Policy Optimization (2025)0.00
• Centralized Cooperative Exploration Policy For Continuous Control Tasks (2023)0.00
• Guided Exploration In Reinforcement Learning Via Monte Carlo Critic Optimization (2022)0.00
• Policy Improvement Via Imitation Of Multiple Oracles (2020)0.00
• Actor-critic Policy Optimization In Partially Observable Multiagent Environments (2018)0.00
• How To Enable Uncertainty Estimation In Proximal Policy Optimization (2022)0.00
• Active Policy Improvement From Multiple Black-box Oracles (2023)0.00
• Conservative Exploration For Policy Optimization Via Off-policy Policy Evaluation (2023)0.00