How To Learn A Useful Critic? Model-based Action-gradient-estimator Policy Optimization

Abstract

Deterministic-policy actor-critic algorithms for continuous control improve the actor by plugging its actions into the critic and ascending the action-value gradient, which is obtained by chaining the actor's Jacobian matrix with the gradient of the critic with respect to input actions. However, instead of gradients, the critic is, typically, only trained to accurately predict expected returns, which, on their own, are useless for policy optimization. In this paper, we propose MAGE, a model-based actor-critic algorithm, grounded in the theory of policy gradients, which explicitly learns the action-value gradient. MAGE backpropagates through the learned dynamics to compute gradient targets in temporal difference learning, leading to a critic tailored for policy improvement. On a set of MuJoCo continuous-control tasks, we demonstrate the efficiency of the algorithm in comparison to model-free and model-based state-of-the-art baselines.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydoro2020how

Related papers

• Value Improved Actor Critic Algorithms (2024)0.00
• Doubly Robust Off-policy Actor-critic Algorithms For Reinforcement Learning (2019)0.00
• Guide Actor-critic For Continuous Control (2017)0.00
• Compatible Gradient Approximations For Actor-critic Algorithms (2024)0.00
• Mean Actor Critic (2017)0.00
• Model Based Meta Learning Of Critics For Policy Gradients (2022)0.00
• Actor Critic Learning Algorithms For Mean-field Control With Moment Neural Networks (2023)0.00
• Revisiting Gaussian Mixture Critics In Off-policy Reinforcement Learning: A Sample-based Approach (2022)0.00