Tensor And Matrix Low-rank Value-function Approximation In Reinforcement Learning

Abstract

Value-function (VF) approximation is a central problem in Reinforcement Learning (RL). Classical non-parametric VF estimation suffers from the curse of dimensionality. As a result, parsimonious parametric models have been adopted to approximate VFs in high-dimensional spaces, with most efforts being focused on linear and neural-network-based approaches. Differently, this paper puts forth a a parsimonious non-parametric approach, where we use stochastic low-rank algorithms to estimate the VF matrix in an online and model-free fashion. Furthermore, as VFs tend to be multi-dimensional, we propose replacing the classical VF matrix representation with a tensor (multi-way array) representation and, then, use the PARAFAC decomposition to design an online model-free tensor low-rank algorithm. Different versions of the algorithms are proposed, their complexity is analyzed, and their performance is assessed numerically using standardized RL environments.

Stats

• citations10
• S2 citations—
• github stars0
• HF likes0
• heat score7.81
• arxiv keyrozada2022tensor

Related papers

• Multilinear Tensor Low-rank Approximation For Policy-gradient Methods In Reinforcement Learning (2025)0.00
• Uncertainty-aware Low-rank Q-matrix Estimation For Deep Reinforcement Learning (2021)0.00
• Viper: Provably Efficient Algorithm For Offline RL With Neural Function Approximation (2023)0.00
• Foresee Then Evaluate: Decomposing Value Estimation With Latent Future Prediction (2021)3.58
• Approximating Two Value Functions Instead Of One: Towards Characterizing A New Family Of Deep Reinforcement Learning Algorithms (2019)0.00
• \(V_{0.5}\): Generalist Value Model As A Prior For Sparse RL Rollouts (2026)0.00
• Value Function Approximations Via Kernel Embeddings For No-regret Reinforcement Learning (2020)0.00
• Offline Reinforcement Learning: Fundamental Barriers For Value Function Approximation (2021)0.00