Learning Good State And Action Representations Via Tensor Decomposition

Abstract

The transition kernel of a continuous-state-action Markov decision process (MDP) admits a natural tensor structure. This paper proposes a tensor-inspired unsupervised learning method to identify meaningful low-dimensional state and action representations from empirical trajectories. The method exploits the MDP's tensor structure by kernelization, importance sampling and low-Tucker-rank approximation. This method can be further used to cluster states and actions respectively and find the best discrete MDP abstraction. We provide sharp statistical error bounds for tensor concentration and the preservation of diffusion distance after embedding. We further prove that the learned state/action abstractions provide accurate approximations to latent block structures if they exist, enabling function approximation in downstream tasks such as policy evaluation.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyni2021learning

Related papers

• Low-dimensional State And Action Representation Learning With MDP Homomorphism Metrics (2021)0.00
• Model Based Multi-agent Reinforcement Learning With Tensor Decompositions (2021)0.00
• Learning The Minimum Action Distance (2025)0.00
• Deepmdp: Learning Continuous Latent Space Models For Representation Learning (2019)0.00
• Learning Markov State Abstractions For Deep Reinforcement Learning (2021)0.00
• On The Geometry Of Reinforcement Learning In Continuous State And Action Spaces (2022)0.00
• Achieving Sample And Computational Efficient Reinforcement Learning By Action Space Reduction Via Grouping (2023)0.00
• Deep Active Inference For Partially Observable Mdps (2020)9.59