On Many-actions Policy Gradient

Abstract

We study the variance of stochastic policy gradients (SPGs) with many action samples per state. We derive a many-actions optimality condition, which determines when many-actions SPG yields lower variance as compared to a single-action agent with proportionally extended trajectory. We propose Model-Based Many-Actions (MBMA), an approach leveraging dynamics models for many-actions sampling in the context of SPG. MBMA addresses issues associated with existing implementations of many-actions SPG and yields lower bias and comparable variance to SPG estimated from states in model-simulated rollouts. We find that MBMA bias and variance structure matches that predicted by theory. As a result, MBMA achieves improved sample efficiency and higher returns on a range of continuous action environments as compared to model-free, many-actions, and model-based on-policy SPG baselines.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keynauman2022on

Related papers

• Variance Reduction For Policy Gradient With Action-dependent Factorized Baselines (2018)0.00
• Marginal Policy Gradients: A Unified Family Of Estimators For Bounded Action Spaces With Applications (2018)0.00
• Settling The Variance Of Multi-agent Policy Gradients (2021)0.00
• Investigation On The Generalization Of The Sampled Policy Gradient Algorithm (2019)0.00
• Variance Reduction Based Partial Trajectory Reuse To Accelerate Policy Gradient Optimization (2022)0.00
• Clipped Action Policy Gradient (2018)0.00
• Action-depedent Control Variates For Policy Optimization Via Stein's Identity (2017)0.00
• MDPGT: Momentum-based Decentralized Policy Gradient Tracking (2021)0.00