Lipschitz Lifelong Reinforcement Learning

Abstract

We consider the problem of knowledge transfer when an agent is facing a series of Reinforcement Learning (RL) tasks. We introduce a novel metric between Markov Decision Processes (MDPs) and establish that close MDPs have close optimal value functions. Formally, the optimal value functions are Lipschitz continuous with respect to the tasks space. These theoretical results lead us to a value-transfer method for Lifelong RL, which we use to build a PAC-MDP algorithm with improved convergence rate. Further, we show the method to experience no negative transfer with high probability. We illustrate the benefits of the method in Lifelong RL experiments.

Stats

• citations12
• S2 citations—
• github stars0
• HF likes0
• heat score8.35
• arxiv keylecarpentier2020lipschitz

Related papers

• Statistical Guarantees For Lifelong Reinforcement Learning Using Pac-bayes Theory (2024)0.00
• Lifelong Reinforcement Learning With Modulating Masks (2022)0.00
• Provably Efficient Multi-task Reinforcement Learning With Model Transfer (2021)0.00
• On Mechanisms For Transfer Using Landmark Value Functions In Multi-task Lifelong Reinforcement Learning (2019)0.00
• Continuous Coordination As A Realistic Scenario For Lifelong Learning (2021)0.00
• Sharing Lifelong Reinforcement Learning Knowledge Via Modulating Masks (2023)0.00
• Reinforcement Learning In The Wild With Maximum Likelihood-based Model Transfer (2023)0.00
• Transfer Q-learning (2022)0.00