Distributionally Robust Off-dynamics Reinforcement Learning: Provable Efficiency With Linear Function Approximation

Abstract

We study off-dynamics Reinforcement Learning (RL), where the policy is trained on a source domain and deployed to a distinct target domain. We aim to solve this problem via online distributionally robust Markov decision processes (DRMDPs), where the learning algorithm actively interacts with the source domain while seeking the optimal performance under the worst possible dynamics that is within an uncertainty set of the source domain's transition kernel. We provide the first study on online DRMDPs with function approximation for off-dynamics RL. We find that DRMDPs' dual formulation can induce nonlinearity, even when the nominal transition kernel is linear, leading to error propagation. By designing a \(d\)-rectangular uncertainty set using the total variation distance, we remove this additional nonlinearity and bypass the error propagation. We then introduce DR-LSVI-UCB, the first provably efficient online DRMDP algorithm for off-dynamics RL with function approximation, and establish

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2024distributionally

Related papers

• Distributionally Robust Offline Reinforcement Learning With Linear Function Approximation (2022)0.00
• Minimax Optimal And Computationally Efficient Algorithms For Distributionally Robust Offline Reinforcement Learning (2024)0.00
• Distributionally Robust Online Markov Game With Linear Function Approximation (2025)0.00
• On The Foundation Of Distributionally Robust Reinforcement Learning (2023)0.00
• Provably Efficient Reinforcement Learning With Linear Function Approximation (2019)11.76
• Robust Adversarial Policy Optimization Under Dynamics Uncertainty (2026)0.00
• Distributional Reinforcement Learning With Linear Function Approximation (2019)0.00
• Distributionally Robust Model-based Offline Reinforcement Learning With Near-optimal Sample Complexity (2022)0.00