Bayesian Risk-averse Q-learning With Streaming Observations

Abstract

We consider a robust reinforcement learning problem, where a learning agent learns from a simulated training environment. To account for the model mis-specification between this training environment and the real environment due to lack of data, we adopt a formulation of Bayesian risk MDP (BRMDP) with infinite horizon, which uses Bayesian posterior to estimate the transition model and impose a risk functional to account for the model uncertainty. Observations from the real environment that is out of the agent's control arrive periodically and are utilized by the agent to update the Bayesian posterior to reduce model uncertainty. We theoretically demonstrate that BRMDP balances the trade-off between robustness and conservativeness, and we further develop a multi-stage Bayesian risk-averse Q-learning algorithm to solve BRMDP with streaming observations from real environment. The proposed algorithm learns a risk-averse yet optimal policy that depends on the availability of real-world obser

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2023bayesian

Related papers

• Online Bayesian Risk-averse Reinforcement Learning (2025)0.00
• Robust Bayesian Dynamic Programming For On-policy Risk-sensitive Reinforcement Learning (2025)0.00
• A Bayesian Approach To Robust Reinforcement Learning (2019)0.00
• Distributional Method For Risk Averse Reinforcement Learning (2023)0.00
• Reinforcement Learning With Markov Risk Measures And Multipattern Risk Approximation (2026)0.00
• Improving Robustness Via Risk Averse Distributional Reinforcement Learning (2020)0.00
• Distributionally Robust Model-based Reinforcement Learning With Large State Spaces (2023)0.00
• Q-learning With Shift-aware Upper Confidence Bound In Non-stationary Reinforcement Learning (2025)0.00