Disentangling Causal Effects For Hierarchical Reinforcement Learning

Abstract

Exploration and credit assignment under sparse rewards are still challenging problems. We argue that these challenges arise in part due to the intrinsic rigidity of operating at the level of actions. Actions can precisely define how to perform an activity but are ill-suited to describe what activity to perform. Instead, causal effects are inherently composable and temporally abstract, making them ideal for descriptive tasks. By leveraging a hierarchy of causal effects, this study aims to expedite the learning of task-specific behavior and aid exploration. Borrowing counterfactual and normality measures from causal literature, we disentangle controllable effects from effects caused by other dynamics of the environment. We propose CEHRL, a hierarchical method that models the distribution of controllable effects using a Variational Autoencoder. This distribution is used by a high-level policy to 1) explore the environment via random effect exploration so that novel effects are continuousl

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keycorcoll2020disentangling

Related papers

• Reducing Action Space For Deep Reinforcement Learning Via Causal Effect Estimation (2025)0.00
• Learning Causal Overhypotheses Through Exploration In Children And Computational Models (2022)0.00
• Learning Nonlinear Causal Reductions To Explain Reinforcement Learning Policies (2025)0.00
• Goal Discovery With Causal Capacity For Efficient Reinforcement Learning (2025)0.00
• ACE : Off-policy Actor-critic With Causality-aware Entropy Regularization (2024)0.00
• Explainable Reinforcement Learning Via A Causal World Model (2023)9.03
• D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based On Causal Discovery And Spurious Correlation Detection (2025)0.00
• Boosting Efficiency In Task-agnostic Exploration Through Causal Knowledge (2024)0.00