Honey, I Shrunk The Actor: A Case Study On Preserving Performance With Smaller Actors In Actor-critic RL

Abstract

Actors and critics in actor-critic reinforcement learning algorithms are functionally separate, yet they often use the same network architectures. This case study explores the performance impact of network sizes when considering actor and critic architectures independently. By relaxing the assumption of architectural symmetry, it is often possible for smaller actors to achieve comparable policy performance to their symmetric counterparts. Our experiments show up to 99% reduction in the number of network weights with an average reduction of 77% over multiple actor-critic algorithms on 9 independent tasks. Given that reducing actor complexity results in a direct reduction of run-time inference cost, we believe configurations of actors and critics are aspects of actor-critic design that deserve to be considered independently, particularly in resource-constrained applications or when deploying multiple actors simultaneously.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymysore2021honey

Related papers

• Studying The Interplay Between The Actor And Critic Representations In Reinforcement Learning (2025)0.00
• Effects Of Sparse Rewards Of Different Magnitudes In The Speed Of Learning Of Model-based Actor Critic Methods (2020)0.00
• Value Improved Actor Critic Algorithms (2024)0.00
• Single-timescale Actor-critic Provably Finds Globally Optimal Policy (2020)0.00
• Actor-critic Or Critic-actor? A Tale Of Two Time Scales (2022)5.84
• Analysis Of A Target-based Actor-critic Algorithm With Linear Function Approximation (2021)0.00
• Actor-attention-critic For Multi-agent Reinforcement Learning (2018)0.00
• Potential Field Guided Actor-critic Reinforcement Learning (2020)0.00