Counterfactual Experience Augmented Off-policy Reinforcement Learning

Abstract

Reinforcement learning control algorithms face significant challenges due to out-of-distribution and inefficient exploration problems. While model-based reinforcement learning enhances the agent's reasoning and planning capabilities by constructing virtual environments, training such virtual environments can be very complex. In order to build an efficient inference model and enhance the representativeness of learning data, we propose the Counterfactual Experience Augmentation (CEA) algorithm. CEA leverages variational autoencoders to model the dynamic patterns of state transitions and introduces randomness to model non-stationarity. This approach focuses on expanding the learning data in the experience pool through counterfactual inference and performs exceptionally well in environments that follow the bisimulation assumption. Environments with bisimulation properties are usually represented by discrete observation and action spaces, we propose a sampling method based on maximum kernel

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylee2025counterfactual

Related papers

• Sample-efficient Reinforcement Learning Via Counterfactual-based Data Augmentation (2020)0.00
• Experience Augmentation: Boosting And Accelerating Off-policy Multi-agent Reinforcement Learning (2020)0.00
• Woulda, Coulda, Shoulda: Counterfactually-guided Policy Search (2018)0.00
• Off-policy Reinforcement Learning With Model-based Exploration Augmentation (2025)0.00
• Using Contrastive Samples For Identifying And Leveraging Possible Causal Relationships In Reinforcement Learning (2022)0.00
• Mocoda: Model-based Counterfactual Data Augmentation (2022)2.26
• Frugal Actor-critic: Sample Efficient Off-policy Deep Reinforcement Learning Using Unique Experiences (2024)0.00
• Autoeg: Automated Experience Grafting For Off-policy Deep Reinforcement Learning (2020)0.00