Reinforcement Learning Of Causal Variables Using Mediation Analysis

Abstract

Many open problems in machine learning are intrinsically related to causality, however, the use of causal analysis in machine learning is still in its early stage. Within a general reinforcement learning setting, we consider the problem of building a general reinforcement learning agent which uses experience to construct a causal graph of the environment, and use this graph to inform its policy. Our approach has three characteristics: First, we learn a simple, coarse-grained causal graph, in which the variables reflect states at many time instances, and the interventions happen at the level of policies, rather than individual actions. Secondly, we use mediation analysis to obtain an optimization target. By minimizing this target, we define the causal variables. Thirdly, our approach relies on estimating conditional expectations rather the familiar expected return from reinforcement learning, and we therefore apply a generalization of Bellman's equations. We show the method can learn a

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keyherlau2020reinforcement

Related papers

• Learning By Doing: An Online Causal Reinforcement Learning Framework With Causal-aware Policy (2024)1.56
• Towards Intervention-centric Causal Reasoning In Learning Agents (2020)0.00
• Causal Reinforcement Learning Using Observational And Interventional Data (2021)0.00
• Resolving Spurious Correlations In Causal Models Of Environments Via Interventions (2020)0.00
• Learning Nonlinear Causal Reductions To Explain Reinforcement Learning Policies (2025)0.00
• Explainable Reinforcement Learning Through A Causal Lens (2019)16.69
• Pessimistic Causal Reinforcement Learning With Mediators For Confounded Offline Data (2024)0.00
• A Roadmap Towards Improving Multi-agent Reinforcement Learning With Causal Discovery And Inference (2025)0.00