The Principle Of Unchanged Optimality In Reinforcement Learning Generalization

Abstract

Several recent papers have examined generalization in reinforcement learning (RL), by proposing new environments or ways to add noise to existing environments, then benchmarking algorithms and model architectures on those environments. We discuss subtle conceptual properties of RL benchmarks that are not required in supervised learning (SL), and also properties that an RL benchmark should possess. Chief among them is one we call the principle of unchanged optimality: there should exist a single \(\pi\) that is optimal across all train and test tasks. In this work, we argue why this principle is important, and ways it can be broken or satisfied due to subtle choices in state representation or model architecture. We conclude by discussing challenges and future lines of research in theoretically analyzing generalization benchmarks.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyirpan2019the

Related papers

• Assessing Generalization In Deep Reinforcement Learning (2018)0.00
• Quantifying Generalization In Reinforcement Learning (2018)0.00
• Measuring And Characterizing Generalization In Deep Reinforcement Learning (2018)9.76
• On The Power Of Pre-training For Generalization In RL: Provable Benefits And Hardness (2022)0.00
• Good Actions Succeed, Bad Actions Generalize: A Case Study On Why RL Generalizes Better (2025)0.00
• Understanding What Affects The Generalization Gap In Visual Reinforcement Learning: Theory And Empirical Evidence (2024)5.84
• Dynamics Generalization Via Information Bottleneck In Deep Reinforcement Learning (2020)0.00
• On Generalization Across Environments In Multi-objective Reinforcement Learning (2025)0.00