Model-invariant State Abstractions For Model-based Reinforcement Learning

Abstract

Accuracy and generalization of dynamics models is key to the success of model-based reinforcement learning (MBRL). As the complexity of tasks increases, so does the sample inefficiency of learning accurate dynamics models. However, many complex tasks also exhibit sparsity in the dynamics, i.e., actions have only a local effect on the system dynamics. In this paper, we exploit this property with a causal invariance perspective in the single-task setting, introducing a new type of state abstraction called \textit\{model-invariance\}. Unlike previous forms of state abstractions, a model-invariance state abstraction leverages causal sparsity over state variables. This allows for compositional generalization to unseen states, something that non-factored forms of state abstractions cannot do. We prove that an optimal policy can be learned over this model-invariance state abstraction and show improved generalization in a simple toy domain. Next, we propose a practical method to approximately

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytomar2021model

Related papers

• An Analysis Of Model-based Reinforcement Learning From Abstracted Observations (2022)0.00
• Fine-grained Causal Dynamics Learning With Quantization For Improving Robustness In Reinforcement Learning (2024)0.00
• Invariant Causal Prediction For Block Mdps (2020)0.00
• Learning Markov State Abstractions For Deep Reinforcement Learning (2021)0.00
• Dynamic Sparsity: Challenging Common Sparsity Assumptions For Learning World Models In Robotic Reinforcement Learning Benchmarks (2025)0.00
• A Relational Intervention Approach For Unsupervised Dynamics Generalization In Model-based Reinforcement Learning (2022)0.00
• Towards Causal Model-based Policy Optimization (2025)0.00
• Learning Robust State Abstractions For Hidden-parameter Block Mdps (2020)0.00