Testing Stationarity And Change Point Detection In Reinforcement Learning

Abstract

We consider offline reinforcement learning (RL) methods in possibly nonstationary environments. Many existing RL algorithms in the literature rely on the stationarity assumption that requires the system transition and the reward function to be constant over time. However, the stationarity assumption is restrictive in practice and is likely to be violated in a number of applications, including traffic signal control, robotics and mobile health. In this paper, we develop a consistent procedure to test the nonstationarity of the optimal Q-function based on pre-collected historical data, without additional online data collection. Based on the proposed test, we further develop a sequential change point detection method that can be naturally coupled with existing state-of-the-art RL methods for policy optimization in nonstationary environments. The usefulness of our method is illustrated by theoretical results, simulation studies, and a real data example from the 2018 Intern Health Study. A

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyli2022testing

Related papers

• Quantifying The Impact Of Non-stationarity In Reinforcement Learning-based Traffic Signal Control (2020)8.35
• Minimum-delay Adaptation In Non-stationary Reinforcement Learning Via Online High-confidence Change-point Detection (2021)0.00
• Regularizing A Model-based Policy Stationary Distribution To Stabilize Offline Reinforcement Learning (2022)0.00
• Demystifying Reinforcement Learning In Time-varying Systems (2022)0.00
• Safe Continual Reinforcement Learning In Non-stationary Environments (2026)12.89
• Online Reinforcement Learning In Non-stationary Context-driven Environments (2023)0.00
• Non-stationary Risk-sensitive Reinforcement Learning: Near-optimal Dynamic Regret, Adaptive Detection, And Separation Design (2022)3.58
• State-constrained Offline Reinforcement Learning (2024)0.00