Improving Exploration In Soft-actor-critic With Normalizing Flows Policies

Abstract

Deep Reinforcement Learning (DRL) algorithms for continuous action spaces are known to be brittle toward hyperparameters as well as \cut\{being\}sample inefficient. Soft Actor Critic (SAC) proposes an off-policy deep actor critic algorithm within the maximum entropy RL framework which offers greater stability and empirical gains. The choice of policy distribution, a factored Gaussian, is motivated by \cut\{chosen due\}its easy re-parametrization rather than its modeling power. We introduce Normalizing Flow policies within the SAC framework that learn more expressive classes of policies than simple factored Gaussians. \cut\{We also present a series of stabilization tricks that enable effective training of these policies in the RL setting.\}We show empirically on continuous grid world tasks that our approach increases stability and is better suited to difficult exploration in sparse reward settings.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyward2019improving

Related papers

• Striving For Simplicity And Performance In Off-policy DRL: Output Normalization And Non-uniform Sampling (2019)0.00
• Discrete And Continuous Action Representation For Practical RL In Video Games (2019)0.00
• DR-SAC: Distributionally Robust Soft Actor-critic For Reinforcement Learning Under Uncertainty (2025)0.00
• Off-policy Actor-critic In An Ensemble: Achieving Maximum General Entropy And Effective Environment Exploration In Deep Reinforcement Learning (2019)0.00
• Max-entropy Reinforcement Learning With Flow Matching And A Case Study On LQR (2025)0.00
• Flowpg: Action-constrained Policy Gradient With Normalizing Flows (2024)0.00
• Boosting Soft Actor-critic: Emphasizing Recent Experience Without Forgetting The Past (2019)0.00
• DSAC: Distributional Soft Actor-critic For Risk-sensitive Reinforcement Learning (2020)7.81