Learning Causal State Representations Of Partially Observable Environments

Abstract

Intelligent agents can cope with sensory-rich environments by learning task-agnostic state abstractions. In this paper, we propose an algorithm to approximate causal states, which are the coarsest partition of the joint history of actions and observations in partially-observable Markov decision processes (POMDP). Our method learns approximate causal state representations from RNNs trained to predict subsequent observations given the history. We demonstrate that these learned state representations are useful for learning policies efficiently in reinforcement learning problems with rich observation spaces. We connect causal states with causal feature sets from the causal inference literature, and also provide theoretical guarantees on the optimality of the continuous version of this causal state representation under Lipschitz assumptions by proving equivalence to bisimulation, a relation between behaviorally equivalent systems. This allows for lower bounds on the optimal value function o

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2019learning

Related papers

• Learning Causal States Under Partial Observability And Perturbation (2025)0.00
• Causal Reinforcement Learning Using Observational And Interventional Data (2021)0.00
• Causally Correct Partial Models For Reinforcement Learning (2020)0.00
• Provable Representation With Efficient Planning For Partial Observable Reinforcement Learning (2023)0.00
• Provably Efficient Reinforcement Learning In Partially Observable Dynamical Systems (2022)0.00
• Learning By Doing: An Online Causal Reinforcement Learning Framework With Causal-aware Policy (2024)1.56
• Invariant Causal Prediction For Block Mdps (2020)0.00
• Learning Markov State Abstractions For Deep Reinforcement Learning (2021)0.00