Query The Agent: Improving Sample Efficiency Through Epistemic Uncertainty Estimation

Abstract

Curricula for goal-conditioned reinforcement learning agents typically rely on poor estimates of the agent's epistemic uncertainty or fail to consider the agents' epistemic uncertainty altogether, resulting in poor sample efficiency. We propose a novel algorithm, Query The Agent (QTA), which significantly improves sample efficiency by estimating the agent's epistemic uncertainty throughout the state space and setting goals in highly uncertain areas. Encouraging the agent to collect data in highly uncertain states allows the agent to improve its estimation of the value function rapidly. QTA utilizes a novel technique for estimating epistemic uncertainty, Predictive Uncertainty Networks (PUN), to allow QTA to assess the agent's uncertainty in all previously observed states. We demonstrate that QTA offers decisive sample efficiency improvements over preexisting methods.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyalverio2022query

Related papers

• Exploration Via Epistemic Value Estimation (2023)2.26
• Estimating Risk And Uncertainty In Deep Reinforcement Learning (2019)0.00
• Enhancing Sample Efficiency In Multi-agent RL With Uncertainty Quantification And Selective Exploration (2025)0.00
• Uncertainty Quantification And Exploration For Reinforcement Learning (2019)6.77
• Aggressive Q-learning With Ensembles: Achieving Both High Sample Efficiency And High Asymptotic Performance (2021)0.00
• On Practical Robust Reinforcement Learning: Practical Uncertainty Set And Double-agent Algorithm (2023)3.58
• Provably Efficient And Agile Randomized Q-learning (2025)0.00
• MEET: A Monte Carlo Exploration-exploitation Trade-off For Buffer Sampling (2022)2.26