Generative Actor-critic: An Off-policy Algorithm Using The Push-forward Model

Abstract

Model-free deep reinforcement learning has achieved great success in many domains, such as video games, recommendation systems and robotic control tasks. In continuous control tasks, widely used policies with Gaussian distributions results in ineffective exploration of environments and limited performance of algorithms in many cases. In this paper, we propose a density-free off-policy algorithm, Generative Actor-Critic(GAC), using the push-forward model to increase the expressiveness of policies, which also includes an entropy-like technique, MMD-entropy regularizer, to balance the exploration and exploitation. Additionnally, we devise an adaptive mechanism to automatically scale this regularizer, which further improves the stability and robustness of GAC. The experiment results show that push-forward policies possess desirable features, such as multi-modality, which can improve the efficiency of exploration and asymptotic performance of algorithms obviously.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypeng2021generative

Related papers

• Guide Actor-critic For Continuous Control (2017)0.00
• Distributional Policy Optimization: An Alternative Approach For Continuous Control (2019)0.00
• Behavior-guided Actor-critic: Improving Exploration Via Learning Policy Behavior Representation For Deep Reinforcement Learning (2021)0.00
• Off-policy Actor-critic In An Ensemble: Achieving Maximum General Entropy And Effective Environment Exploration In Deep Reinforcement Learning (2019)0.00
• Revisiting Gaussian Mixture Critics In Off-policy Reinforcement Learning: A Sample-based Approach (2022)0.00
• Boosting Exploration In Actor-critic Algorithms By Incentivizing Plausible Novel States (2022)5.24
• Wasserstein Barycenter Soft Actor-critic (2025)0.00
• PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithm (2023)0.00