Online Reinforcement Learning In Non-stationary Context-driven Environments

Abstract

We study online reinforcement learning (RL) in non-stationary environments, where a time-varying exogenous context process affects the environment dynamics. Online RL is challenging in such environments due to "catastrophic forgetting" (CF). The agent tends to forget prior knowledge as it trains on new experiences. Prior approaches to mitigate this issue assume task labels (which are often not available in practice), employ brittle regularization heuristics, or use off-policy methods that suffer from instability and poor performance. We present Locally Constrained Policy Optimization (LCPO), an online RL approach that combats CF by anchoring policy outputs on old experiences while optimizing the return on current experiences. To perform this anchoring, LCPO locally constrains policy optimization using samples from experiences that lie outside of the current context distribution. We evaluate LCPO in Mujoco, classic control and computer systems environments with a variety of synthetic

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhamadanian2023online

Related papers

• Dynamics-adaptive Continual Reinforcement Learning Via Progressive Contextualization (2022)7.16
• Think Outside The Policy: In-context Steered Policy Optimization (2025)0.00
• Demystifying Reinforcement Learning In Time-varying Systems (2022)0.00
• Unified Policy Optimization For Continuous-action Reinforcement Learning In Non-stationary Tasks And Games (2022)2.26
• Compositional Conservatism: A Transductive Approach In Offline Reinforcement Learning (2024)1.81
• Context-based Soft Actor Critic For Environments With Non-stationary Dynamics (2021)0.00
• Context-aware Safe Reinforcement Learning For Non-stationary Environments (2021)9.76
• Learning A Subspace Of Policies For Online Adaptation In Reinforcement Learning (2021)0.00