Learning To Stabilize Online Reinforcement Learning In Unbounded State Spaces

Abstract

In many reinforcement learning (RL) applications, we want policies that reach desired states and then keep the controlled system within an acceptable region around the desired states over an indefinite period of time. This latter objective is called stability and is especially important when the state space is unbounded, such that the states can be arbitrarily far from each other and the agent can drift far away from the desired states. For example, in stochastic queuing networks, where queues of waiting jobs can grow without bound, the desired state is all-zero queue lengths. Here, a stable policy ensures queue lengths are finite while an optimal policy minimizes queue lengths. Since an optimal policy is also stable, one would expect that RL algorithms would implicitly give us stable policies. However, in this work, we find that deep RL algorithms that directly minimize the distance to the desired state during online training often result in unstable policies, i.e., policies that drif

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypavse2023learning

Related papers

• Stable Reinforcement Learning With Unbounded State Space (2020)0.00
• Taming "data-hungry" Reinforcement Learning? Stability In Continuous State-action Spaces (2024)2.26
• Regularizing A Model-based Policy Stationary Distribution To Stabilize Offline Reinforcement Learning (2022)0.00
• Moments Matter:stabilizing Policy Optimization Using Return Distributions (2026)0.00
• Learning A Subspace Of Policies For Online Adaptation In Reinforcement Learning (2021)0.00
• Reinforcement Learning With Fast Stabilization In Linear Dynamical Systems (2020)0.00
• Goal-conditioned Offline Reinforcement Learning Through State Space Partitioning (2023)2.26
• Demystifying Reinforcement Learning In Time-varying Systems (2022)0.00