Adaptive Regularization Of Representation Rank As An Implicit Constraint Of Bellman Equation

Abstract

Representation rank is an important concept for understanding the role of Neural Networks (NNs) in Deep Reinforcement learning (DRL), which measures the expressive capacity of value networks. Existing studies focus on unboundedly maximizing this rank; nevertheless, that approach would introduce overly complex models in the learning, thus undermining performance. Hence, fine-tuning representation rank presents a challenging and crucial optimization problem. To address this issue, we find a guiding principle for adaptive control of the representation rank. We employ the Bellman equation as a theoretical foundation and derive an upper bound on the cosine similarity of consecutive state-action pairs representations of value networks. We then leverage this upper bound to propose a novel regularizer, namely BEllman Equation-based automatic rank Regularizer (BEER). This regularizer adaptively regularizes the representation rank, thus improving the DRL agent's performance. We first validate th

Stats

• citations0
• S2 citations—
• github stars1
• HF likes0
• heat score0.60
• arxiv keyhe2024adaptive

Related papers

• Frustratingly Easy Regularization On Representation Can Boost Deep Reinforcement Learning (2022)0.00
• Implicit Under-parameterization Inhibits Data-efficient Deep Reinforcement Learning (2020)0.00
• Spectral Bellman Method: Unifying Representation And Exploration In RL (2025)0.00
• Exploiting Structure In Offline Multi-agent RL: The Benefits Of Low Interaction Rank (2024)0.00
• Locally Constrained Representations In Reinforcement Learning (2022)0.00
• On The Generalization Of Representations In Reinforcement Learning (2022)0.00
• DR3: Value-based Deep Reinforcement Learning Requires Explicit Regularization (2021)0.00
• Regularization Matters In Policy Optimization (2019)2.68