Policy Optimization For Continuous Reinforcement Learning

Abstract

We study reinforcement learning (RL) in the setting of continuous time and space, for an infinite horizon with a discounted objective and the underlying dynamics driven by a stochastic differential equation. Built upon recent advances in the continuous approach to RL, we develop a notion of occupation time (specifically for a discounted objective), and show how it can be effectively used to derive performance-difference and local-approximation formulas. We further extend these results to illustrate their applications in the PG (policy gradient) and TRPO/PPO (trust region policy optimization/ proximal policy optimization) methods, which have been familiar and powerful tools in the discrete RL setting but under-developed in continuous RL. Through numerical experiments, we demonstrate the effectiveness and advantages of our approach.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyzhao2023policy

Related papers

• Unified Policy Optimization For Continuous-action Reinforcement Learning In Non-stationary Tasks And Games (2022)2.26
• Deterministic Policy Gradient For Reinforcement Learning With Continuous Time And State (2025)0.00
• Policy Optimization In A Noisy Neighborhood: On Return Landscapes In Continuous Control (2023)0.00
• Policy Optimization Over General State And Action Spaces (2022)0.00
• Conservative Optimistic Policy Optimization Via Multiple Importance Sampling (2021)0.00
• Taming "data-hungry" Reinforcement Learning? Stability In Continuous State-action Spaces (2024)2.26
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00
• Conservative Exploration For Policy Optimization Via Off-policy Policy Evaluation (2023)0.00