Finite-time Convergence And Sample Complexity Of Actor-critic Multi-objective Reinforcement Learning

Abstract

Reinforcement learning with multiple, potentially conflicting objectives is pervasive in real-world applications, while this problem remains theoretically under-explored. This paper tackles the multi-objective reinforcement learning (MORL) problem and introduces an innovative actor-critic algorithm named MOAC which finds a policy by iteratively making trade-offs among conflicting reward signals. Notably, we provide the first analysis of finite-time Pareto-stationary convergence and corresponding sample complexity in both discounted and average reward settings. Our approach has two salient features: (a) MOAC mitigates the cumulative estimation bias resulting from finding an optimal common gradient descent direction out of stochastic samples. This enables provable convergence rate and sample complexity guarantees independent of the number of objectives; (b) With proper momentum coefficient, MOAC initializes the weights of individual policy gradients using samples from the environment, in

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhou2024finite

Related papers

• A Scale-independent Multi-objective Reinforcement Learning With Convergence Analysis (2023)2.26
• Actor-critic Algorithms For Constrained Multi-agent Reinforcement Learning (2019)0.00
• Global Convergence Of Two-timescale Actor-critic For Solving Linear Quadratic Regulator (2022)4.52
• A Sharper Global Convergence Analysis For Average Reward Reinforcement Learning Via An Actor-critic Approach (2024)0.00
• A Finite Time Analysis Of Two Time-scale Actor Critic Methods (2020)0.00
• On The Sample Complexity Of Actor-critic Method For Reinforcement Learning With Function Approximation (2019)11.49
• Finite-time Analysis Of Fully Decentralized Single-timescale Actor-critic (2022)0.00
• Improving Sample Complexity Bounds For (natural) Actor-critic Algorithms (2020)0.00