Spectral Bellman Method: Unifying Representation And Exploration In RL

Abstract

Representation learning is critical to the empirical and theoretical success of reinforcement learning. However, many existing methods are induced from model-learning aspects, misaligning them with the RL task in hand. This work introduces the Spectral Bellman Method, a novel framework derived from the Inherent Bellman Error (IBE) condition. It aligns representation learning with the fundamental structure of Bellman updates across a \textit\{space\} of possible value functions, making it directly suited for value-based RL. Our key insight is a fundamental spectral relationship: under the zero-IBE condition, the transformation of a \textit\{distribution\} of value functions by the Bellman operator is intrinsically linked to the feature covariance structure. This connection yields a new, theoretically-grounded objective for learning state-action features that capture this Bellman-aligned covariance, requiring only a simple modification to existing algorithms. We demonstrate that our lear

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keynabati2025spectral

Related papers

• Spectral Representation-based Reinforcement Learning (2025)0.00
• Spectral Decomposition Representation For Reinforcement Learning (2022)0.00
• Nonparametric Bellman Mappings For Value Iteration In Distributed Reinforcement Learning (2025)0.00
• Nonparametric Bellman Mappings For Reinforcement Learning: Application To Robust Adaptive Filtering (2024)6.34
• The Value-improvement Path: Towards Better Representations For Reinforcement Learning (2020)6.77
• Balancing Interpretability And Performance In Reinforcement Learning: An Adaptive Spectral Based Linear Approach (2025)0.00
• Simplifying Model-based RL: Learning Representations, Latent-space Models, And Policies With One Objective (2022)0.00
• Representation-driven Reinforcement Learning (2023)0.00