Unified Policy Optimization For Continuous-action Reinforcement Learning In Non-stationary Tasks And Games

Abstract

This paper addresses policy learning in non-stationary environments and games with continuous actions. Rather than the classical reward maximization mechanism, inspired by the ideas of follow-the-regularized-leader (FTRL) and mirror descent (MD) update, we propose a no-regret style reinforcement learning algorithm PORL for continuous action tasks. We prove that PORL has a last-iterate convergence guarantee, which is important for adversarial and cooperative games. Empirical studies show that, in stationary environments such as MuJoCo locomotion controlling tasks, PORL performs equally well as, if not better than, the soft actor-critic (SAC) algorithm; in non-stationary environments including dynamical environments, adversarial training, and competitive games, PORL is superior to SAC in both a better final policy performance and a more stable training process.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyqin2022unified

Related papers

• Policy Optimization For Continuous Reinforcement Learning (2023)2.26
• Online Reinforcement Learning In Non-stationary Context-driven Environments (2023)0.00
• Actor-critic Policy Optimization In Partially Observable Multiagent Environments (2018)0.00
• Policy Optimization In A Noisy Neighborhood: On Return Landscapes In Continuous Control (2023)0.00
• Continuously Discovering Novel Strategies Via Reward-switching Policy Optimization (2022)0.00
• Faster Last-iterate Convergence Of Policy Optimization In Zero-sum Markov Games (2022)0.00
• ANO: A Principled Approach To Robust Policy Optimization (2026)0.00
• Policy Optimization For Markov Games: Unified Framework And Faster Convergence (2022)0.00