Quantile Q-learning: Revisiting Offline Extreme Q-learning With Quantile Regression

Abstract

Offline reinforcement learning (RL) enables policy learning from fixed datasets without further environment interaction, making it particularly valuable in high-risk or costly domains. Extreme \(Q\)-Learning (XQL) is a recent offline RL method that models Bellman errors using the Extreme Value Theorem, yielding strong empirical performance. However, XQL and its stabilized variant MXQL suffer from notable limitations: both require extensive hyperparameter tuning specific to each dataset and domain, and also exhibit instability during training. To address these issues, we proposed a principled method to estimate the temperature coefficient \(\beta\) via quantile regression under mild assumptions. To further improve training stability, we introduce a value regularization technique with mild generalization, inspired by recent advances in constrained value learning. Experimental results demonstrate that the proposed algorithm achieves competitive or superior performance across a range of be

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygao2025quantile

Related papers

• Stabilizing Extreme Q-learning By Maclaurin Expansion (2024)0.00
• Model-based Offline Reinforcement Learning With Lower Expectile Q-learning (2024)0.00
• PIQL: Projective Implicit Q-learning With Support Constraint For Offline Reinforcement Learning (2025)0.00
• ACL-QL: Adaptive Conservative Level In Q-learning For Offline Reinforcement Learning (2024)0.00
• Cal-ql: Calibrated Offline RL Pre-training For Efficient Online Fine-tuning (2023)2.26
• Mildly Conservative Q-learning For Offline Reinforcement Learning (2022)0.00
• Emaq: Expected-max Q-learning Operator For Simple Yet Effective Offline And Online RL (2020)0.00
• Distributional Reinforcement Learning With Quantile Regression (2017)19.20