Transient Non-stationarity And Generalisation In Deep Reinforcement Learning

Abstract

Non-stationarity can arise in Reinforcement Learning (RL) even in stationary environments. For example, most RL algorithms collect new data throughout training, using a non-stationary behaviour policy. Due to the transience of this non-stationarity, it is often not explicitly addressed in deep RL and a single neural network is continually updated. However, we find evidence that neural networks exhibit a memory effect where these transient non-stationarities can permanently impact the latent representation and adversely affect generalisation performance. Consequently, to improve generalisation of deep RL agents, we propose Iterated Relearning (ITER). ITER augments standard RL training by repeated knowledge transfer of the current policy into a freshly initialised network, which thereby experiences less non-stationarity during training. Experimentally, we show that ITER improves performance on the challenging generalisation benchmarks ProcGen and Multiroom.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyigl2020transient

Related papers

• Demystifying Reinforcement Learning In Time-varying Systems (2022)0.00
• Model-agnostic Solutions For Deep Reinforcement Learning In Non-ergodic Contexts (2026)0.00
• Illuminating Generalization In Deep Reinforcement Learning Through Procedural Level Generation (2018)0.00
• Temporal Disentanglement Of Representations For Improved Generalisation In Reinforcement Learning (2022)0.00
• Measuring And Characterizing Generalization In Deep Reinforcement Learning (2018)9.76
• Generalization In Transfer Learning (2019)0.00
• Dynamics Generalization Via Information Bottleneck In Deep Reinforcement Learning (2020)0.00
• Assessing Generalization In Deep Reinforcement Learning (2018)0.00