Uncertainty-aware Low-rank Q-matrix Estimation For Deep Reinforcement Learning

Abstract

Value estimation is one key problem in Reinforcement Learning. Albeit many successes have been achieved by Deep Reinforcement Learning (DRL) in different fields, the underlying structure and learning dynamics of value function, especially with complex function approximation, are not fully understood. In this paper, we report that decreasing rank of \(Q\)-matrix widely exists during learning process across a series of continuous control tasks for different popular algorithms. We hypothesize that the low-rank phenomenon indicates the common learning dynamics of \(Q\)-matrix from stochastic high dimensional space to smooth low dimensional space. Moreover, we reveal a positive correlation between value matrix rank and value estimation uncertainty. Inspired by above evidence, we propose a novel Uncertainty-Aware Low-rank Q-matrix Estimation (UA-LQE) algorithm as a general framework to facilitate the learning of value function. Through quantifying the uncertainty of state-action value estima

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysang2021uncertainty

Related papers

• Tensor And Matrix Low-rank Value-function Approximation In Reinforcement Learning (2022)7.81
• Estimating Risk And Uncertainty In Deep Reinforcement Learning (2019)0.00
• Uncertainty Quantification And Exploration For Reinforcement Learning (2019)6.77
• Utilizing Maximum Mean Discrepancy Barycenter For Propagating The Uncertainty Of Value Functions In Reinforcement Learning (2024)0.00
• Approximating Two Value Functions Instead Of One: Towards Characterizing A New Family Of Deep Reinforcement Learning Algorithms (2019)0.00
• An Information-theoretic Optimality Principle For Deep Reinforcement Learning (2017)0.00
• Model-based Offline Reinforcement Learning With Lower Expectile Q-learning (2024)0.00
• Value-distributional Model-based Reinforcement Learning (2023)1.56