Frustratingly Easy Regularization On Representation Can Boost Deep Reinforcement Learning

Abstract

Deep reinforcement learning (DRL) gives the promise that an agent learns good policy from high-dimensional information, whereas representation learning removes irrelevant and redundant information and retains pertinent information. In this work, we demonstrate that the learned representation of the \(Q\)-network and its target \(Q\)-network should, in theory, satisfy a favorable distinguishable representation property. Specifically, there exists an upper bound on the representation similarity of the value functions of two adjacent time steps in a typical DRL setting. However, through illustrative experiments, we show that the learned DRL agent may violate this property and lead to a sub-optimal policy. Therefore, we propose a simple yet effective regularizer called Policy Evaluation with Easy Regularization on Representation (PEER), which aims to maintain the distinguishable representation property via explicit regularization on internal representations. And we provide the convergence

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhe2022frustratingly

Related papers

• Deep Reinforcement Learning With Decorrelation (2019)0.00
• Regularization Matters In Policy Optimization (2019)2.68
• How Transferable Are The Representations Learned By Deep Q Agents? (2020)0.00
• Representation-driven Reinforcement Learning (2023)0.00
• Locally Constrained Representations In Reinforcement Learning (2022)0.00
• Rethinking Model-based, Policy-based, And Value-based Reinforcement Learning Via The Lens Of Representation Complexity (2023)2.26
• Adaptive Regularization Of Representation Rank As An Implicit Constraint Of Bellman Equation (2024)0.60
• Erl-re\(^2\): Efficient Evolutionary Reinforcement Learning With Shared State Representation And Individual Policy Representation (2022)0.00