Matrix Estimation For Offline Reinforcement Learning With Low-rank Structure

Abstract

We consider offline Reinforcement Learning (RL), where the agent does not interact with the environment and must rely on offline data collected using a behavior policy. Previous works provide policy evaluation guarantees when the target policy to be evaluated is covered by the behavior policy, that is, state-action pairs visited by the target policy must also be visited by the behavior policy. We show that when the MDP has a latent low-rank structure, this coverage condition can be relaxed. Building on the connection to weighted matrix completion with non-uniform observations, we propose an offline policy evaluation algorithm that leverages the low-rank structure to estimate the values of uncovered state-action pairs. Our algorithm does not require a known feature representation, and our finite-sample error bound involves a novel discrepancy measure quantifying the discrepancy between the behavior and target policies in the spectral space. We provide concrete examples where our algorit

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyxi2023matrix

Related papers

• Model-free Low-rank Reinforcement Learning Via Leveraged Entry-wise Matrix Estimation (2024)0.00
• Behavior Estimation From Multi-source Data For Offline Reinforcement Learning (2022)2.26
• Projected State-action Balancing Weights For Offline Reinforcement Learning (2021)0.00
• Multilinear Tensor Low-rank Approximation For Policy-gradient Methods In Reinforcement Learning (2025)0.00
• Offline Policy Evaluation For Reinforcement Learning With Adaptively Collected Data (2023)0.00
• Constrained Latent Action Policies For Model-based Offline Reinforcement Learning (2024)0.00
• Regularizing A Model-based Policy Stationary Distribution To Stabilize Offline Reinforcement Learning (2022)0.00
• Optimal Single-policy Sample Complexity And Transient Coverage For Average-reward Offline RL (2025)0.00