Band-limited Soft Actor Critic Model

Abstract

Soft Actor Critic (SAC) algorithms show remarkable performance in complex simulated environments. A key element of SAC networks is entropy regularization, which prevents the SAC actor from optimizing against fine grained features, oftentimes transient, of the state-action value function. This results in better sample efficiency during early training. We take this idea one step further by artificially bandlimiting the target critic spatial resolution through the addition of a convolutional filter. We derive the closed form solution in the linear case and show that bandlimiting reduces the interdependency between the low and high frequency components of the state-action value approximation, allowing the critic to learn faster. In experiments, the bandlimited SAC outperformed the classic twin-critic SAC in a number of Gym environments, and displayed more stability in returns. We derive novel insights about SAC by adding a stochastic noise disturbance, a technique that is increasingly bein

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keycampo2020band

Related papers

• DSAC-C: Constrained Maximum Entropy For Robust Discrete Soft-actor Critic (2023)0.00
• Improved Soft Actor-critic: Mixing Prioritized Off-policy Samples With On-policy Experience (2021)0.00
• SARC: Soft Actor Retrospective Critic (2023)0.00
• Soft Actor-critic With Inhibitory Networks For Faster Retraining (2022)0.00
• Boosting Soft Actor-critic: Emphasizing Recent Experience Without Forgetting The Past (2019)0.00
• Revisiting Discrete Soft Actor-critic (2022)0.00
• DSAC: Distributional Soft Actor-critic For Risk-sensitive Reinforcement Learning (2020)7.81
• Context-based Soft Actor Critic For Environments With Non-stationary Dynamics (2021)0.00