Representations For Stable Off-policy Reinforcement Learning

Abstract

Reinforcement learning with function approximation can be unstable and even divergent, especially when combined with off-policy learning and Bellman updates. In deep reinforcement learning, these issues have been dealt with empirically by adapting and regularizing the representation, in particular with auxiliary tasks. This suggests that representation learning may provide a means to guarantee stability. In this paper, we formally show that there are indeed nontrivial state representations under which the canonical TD algorithm is stable, even when learning off-policy. We analyze representation learning schemes that are based on the transition matrix of a policy, such as proto-value functions, along three axes: approximation error, stability, and ease of estimation. In the most general case, we show that a Schur basis provides convergence guarantees, but is difficult to estimate from samples. For a fixed reward function, we find that an orthogonal basis of the corresponding Krylov subs

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyghosh2020representations

Related papers

• Stable Offline Value Function Learning With Bisimulation-based Representations (2024)0.00
• Towards A Better Understanding Of Representation Dynamics Under Td-learning (2023)0.00
• Locally Constrained Representations In Reinforcement Learning (2022)0.00
• On The Generalization Of Representations In Reinforcement Learning (2022)0.00
• When Is Realizability Sufficient For Off-policy Reinforcement Learning? (2022)0.00
• Distributionally Robust Off-dynamics Reinforcement Learning: Provable Efficiency With Linear Function Approximation (2024)0.00
• The Optimal Approximation Factors In Misspecified Off-policy Value Function Estimation (2023)0.00
• SBEED: Convergent Reinforcement Learning With Nonlinear Function Approximation (2017)0.00