Measuring And Characterizing Generalization In Deep Reinforcement Learning

Abstract

Deep reinforcement-learning methods have achieved remarkable performance on challenging control tasks. Observations of the resulting behavior give the impression that the agent has constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, and propose several definitions based on an agent's performance in on-policy, off-policy, and unreachable states. We propose a set of practical methods for evaluating agents with these definitions of generalization. We demonstrate these techniques on a common benchmark task for deep RL, and we show that the learned networks make poor decisions for states that differ only slightly from on-policy states, even though those states are not selected adversarially. Taken together, these results call into question the extent to which deep Q-networks learn generalized representations, and suggest that more experimentation and analysis is necessary before claims of representation lea

Stats

• citations19
• S2 citations—
• github stars0
• HF likes0
• heat score9.76
• arxiv keywitty2018measuring

Related papers

• Assessing Generalization In Deep Reinforcement Learning (2018)0.00
• A Survey Analyzing Generalization In Deep Reinforcement Learning (2024)0.00
• On The Generalization Of Representations In Reinforcement Learning (2022)0.00
• Generalizing Skills With Semi-supervised Reinforcement Learning (2016)0.00
• The Principle Of Unchanged Optimality In Reinforcement Learning Generalization (2019)0.00
• Quantifying Generalization In Reinforcement Learning (2018)0.00
• Understanding What Affects The Generalization Gap In Visual Reinforcement Learning: Theory And Empirical Evidence (2024)5.84
• Dynamics Generalization Via Information Bottleneck In Deep Reinforcement Learning (2020)0.00