Posterior Sampling With Delayed Feedback For Reinforcement Learning With Linear Function Approximation

Abstract

Recent studies in reinforcement learning (RL) have made significant progress by leveraging function approximation to alleviate the sample complexity hurdle for better performance. Despite the success, existing provably efficient algorithms typically rely on the accessibility of immediate feedback upon taking actions. The failure to account for the impact of delay in observations can significantly degrade the performance of real-world systems due to the regret blow-up. In this work, we tackle the challenge of delayed feedback in RL with linear function approximation by employing posterior sampling, which has been shown to empirically outperform the popular UCB algorithms in a wide range of regimes. We first introduce Delayed-PSVI, an optimistic value-based algorithm that effectively explores the value function space via noise perturbation with posterior sampling. We provide the first analysis for posterior sampling algorithms with delayed feedback in RL and show our algorithm achieves \

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykuang2023posterior

Related papers

• Prior-dependent Analysis Of Posterior Sampling Reinforcement Learning With Function Approximation (2024)0.00
• Provably Efficient Reinforcement Learning With Linear Function Approximation (2019)11.76
• Uniform-pac Bounds For Reinforcement Learning With Linear Function Approximation (2021)0.00
• Posterior Sampling For Competitive RL: Function Approximation And Partial Observation (2023)0.00
• Provably Efficient Iterated Cvar Reinforcement Learning With Function Approximation And Human Feedback (2023)0.00
• Nonstationary Reinforcement Learning With Linear Function Approximation (2020)0.00
• Online Sub-sampling For Reinforcement Learning With General Function Approximation (2021)0.00
• Pessimistic Nonlinear Least-squares Value Iteration For Offline Reinforcement Learning (2023)0.00