Batch Size-invariance For Policy Optimization

Abstract

We say an algorithm is batch size-invariant if changes to the batch size can largely be compensated for by changes to other hyperparameters. Stochastic gradient descent is well-known to have this property at small batch sizes, via the learning rate. However, some policy optimization algorithms (such as PPO) do not have this property, because of how they control the size of policy updates. In this work we show how to make these algorithms batch size-invariant. Our key insight is to decouple the proximal policy (used for controlling policy updates) from the behavior policy (used for off-policy corrections). Our experiments help explain why these algorithms work, and additionally show how they can make more efficient use of stale data.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhilton2021batch

Related papers

• Proximal Policy Optimization Algorithms (2017)0.00
• A Parametric Class Of Approximate Gradient Updates For Policy Optimization (2022)0.00
• Model Selection In Batch Policy Optimization (2021)0.00
• Gradient Informed Proximal Policy Optimization (2023)5.15
• Revisiting Design Choices In Proximal Policy Optimization (2020)0.00
• Black-box Combinatorial Optimization With Order-invariant Reinforcement Learning (2025)0.00
• Off-policy Policy Gradient With State Distribution Correction (2019)0.00
• Batch Policy Learning In Average Reward Markov Decision Processes (2020)0.00