Studying The Interplay Between The Actor And Critic Representations In Reinforcement Learning
2025 · Samuel Garcin, Trevor McInroe, Pablo Samuel Castro, et al.
Abstract
Extracting relevant information from a stream of high-dimensional observations is a central challenge for deep reinforcement learning agents. Actor-critic algorithms add further complexity to this challenge, as it is often unclear whether the same information will be relevant to both the actor and the critic. To this end, we here explore the principles that underlie effective representations for the actor and for the critic in on-policy algorithms. We focus our study on understanding whether the actor and critic will benefit from separate, rather than shared, representations. Our primary finding is that when separated, the representations for the actor and critic systematically specialise in extracting different types of information from the environment -- the actor's representation tends to focus on action-relevant information, while the critic's representation specialises in encoding value and dynamics information. We conduct a rigourous empirical study to understand how different re
Authors
(none)
Tags
Stats
Related papers
- Actor-attention-critic For Multi-agent Reinforcement Learning (2018)0.00
- Contrasting Centralized And Decentralized Critics In Multi-agent Reinforcement Learning (2021)0.00
- Wasserstein Flow Meets Replicator Dynamics: A Mean-field Analysis Of Representation Learning In Actor-critic (2021)0.00
- Efficient Exploration In Deep Reinforcement Learning: A Novel Bayesian Actor-critic Algorithm (2024)0.00
- Multi-agent Actor-critic For Mixed Cooperative-competitive Environments (2017)0.00
- Effects Of Sparse Rewards Of Different Magnitudes In The Speed Of Learning Of Model-based Actor Critic Methods (2020)0.00
- Visualizing Critic Match Loss Landscapes For Interpretation Of Online Reinforcement Learning Control Algorithms (2026)0.00
- Stackelberg Actor-critic: Game-theoretic Reinforcement Learning Algorithms (2021)0.00