Woulda, Coulda, Shoulda: Counterfactually-guided Policy Search

Abstract

Learning policies on data synthesized by models can in principle quench the thirst of reinforcement learning algorithms for large amounts of real experience, which is often costly to acquire. However, simulating plausible experience de novo is a hard problem for many complex environments, often resulting in biases for model-based policy evaluation and search. Instead of de novo synthesis of data, here we assume logged, real experience and model alternative outcomes of this experience under counterfactual actions, actions that were not actually taken. Based on this, we propose the Counterfactually-Guided Policy Search (CF-GPS) algorithm for learning policies in POMDPs from off-policy experience. It leverages structural causal models for counterfactual evaluation of arbitrary policies on individual off-policy episodes. CF-GPS can improve on vanilla model-based RL algorithms by making use of available logged data to de-bias model predictions. In contrast to off-policy algorithms based on

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybuesing2018woulda

Related papers

• Learning Efficient And Effective Exploration Policies With Counterfactual Meta Policy (2019)0.00
• Counterfactual Experience Augmented Off-policy Reinforcement Learning (2025)0.00
• Towards Causal Model-based Policy Optimization (2025)0.00
• Gradient-aware Model-based Policy Search (2019)6.77
• Conservative Exploration For Policy Optimization Via Off-policy Policy Evaluation (2023)0.00
• Think Outside The Policy: In-context Steered Policy Optimization (2025)0.00
• Efficient Model-based Reinforcement Learning Through Optimistic Policy Search And Planning (2020)0.00
• Reasoning About Unforeseen Possibilities During Policy Learning (2018)0.00