Visualizing Critic Match Loss Landscapes For Interpretation Of Online Reinforcement Learning Control Algorithms

Abstract

Reinforcement learning has proven its power on various occasions. However, its performance is not always guaranteed when system dynamics change. Instead, it largely relies on users' empirical experience. For reinforcement learning algorithms with an actor-critic structure, the critic neural network reflects the approximation and optimization process in the RL algorithm. Analyzing the performance of the critic neural network helps to understand the mechanism of the algorithm. To support systematic interpretation of such algorithms in dynamic control problems, this work proposes a critic match loss landscape visualization method for online reinforcement learning. The method constructs a loss landscape by projecting recorded critic parameter trajectories onto a low-dimensional linear subspace. The critic match loss is evaluated over the projected parameter grid using fixed reference state samples and temporal-difference targets. This yields a three-dimensional loss surface together with a

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2026visualizing

Related papers

• Studying The Interplay Between The Actor And Critic Representations In Reinforcement Learning (2025)0.00
• Guided Exploration In Reinforcement Learning Via Monte Carlo Critic Optimization (2022)0.00
• Attraction-repulsion Actor-critic For Continuous Control Reinforcement Learning (2019)0.00
• Adviser-actor-critic: Eliminating Steady-state Error In Reinforcement Learning Control (2025)0.00
• Stackelberg Actor-critic: Game-theoretic Reinforcement Learning Algorithms (2021)0.00
• Contrasting Centralized And Decentralized Critics In Multi-agent Reinforcement Learning (2021)0.00
• Langevin Soft Actor-critic: Efficient Exploration Through Uncertainty-driven Critic Learning (2025)0.00
• How To Learn A Useful Critic? Model-based Action-gradient-estimator Policy Optimization (2020)0.00