Time Discretization-invariant Safe Action Repetition For Policy Gradient Methods

Abstract

In reinforcement learning, continuous time is often discretized by a time scale \(\delta\), to which the resulting performance is known to be highly sensitive. In this work, we seek to find a \(\delta\)-invariant algorithm for policy gradient (PG) methods, which performs well regardless of the value of \(\delta\). We first identify the underlying reasons that cause PG methods to fail as \(\delta \to 0\), proving that the variance of the PG estimator can diverge to infinity in stochastic environments under a certain assumption of stochasticity. While durative actions or action repetition can be employed to have \(\delta\)-invariance, previous action repetition methods cannot immediately react to unexpected situations in stochastic environments. We thus propose a novel \(\delta\)-invariant method named Safe Action Repetition (SAR) applicable to any existing PG algorithm. SAR can handle the stochasticity of environments by adaptively reacting to changes in states during action repetition.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypark2021time

Related papers

• Smoothing Policies And Safe Policy Gradients (2019)7.50
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00
• Variance Reduction Based Partial Trajectory Reuse To Accelerate Policy Gradient Optimization (2022)0.00
• On The Convergence And Sample Efficiency Of Variance-reduced Policy Gradient Method (2021)0.00
• Deterministic Policy Gradient For Reinforcement Learning With Continuous Time And State (2025)0.00
• Segmenting Action-value Functions Over Time-scales In SARSA Via Td(\(\delta\)) (2024)0.00
• Zeroth-order Deterministic Policy Gradient (2020)0.00
• Reusing Trajectories In Policy Gradients Enables Fast Convergence (2025)0.00