GINO-Q: Learning An Asymptotically Optimal Index Policy For Restless Multi-armed Bandits

Abstract

The restless multi-armed bandit (RMAB) framework is a popular model with applications across a wide variety of fields. However, its solution is hindered by the exponentially growing state space (with respect to the number of arms) and the combinatorial action space, making traditional reinforcement learning methods infeasible for large-scale instances. In this paper, we propose GINO-Q, a three-timescale stochastic approximation algorithm designed to learn an asymptotically optimal index policy for RMABs. GINO-Q mitigates the curse of dimensionality by decomposing the RMAB into a series of subproblems, each with the same dimension as a single arm, ensuring that complexity increases linearly with the number of arms. Unlike recently developed Whittle-index-based algorithms, GINO-Q does not require RMABs to be indexable, enhancing its flexibility and applicability. Our experimental results demonstrate that GINO-Q consistently learns near-optimal policies, even for non-indexable RMABs where

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2024gino

Related papers

• Q-learning Lagrange Policies For Multi-action Restless Bandits (2021)8.35
• Finite-time Analysis Of Whittle Index Based Q-learning For Restless Multi-armed Bandits With Neural Network Function Approximation (2023)0.00
• Towards A Pretrained Model For Restless Bandits Via Multi-arm Generalization (2023)0.00
• Provably Efficient Reinforcement Learning For Adversarial Restless Multi-armed Bandits With Unknown Transitions And Bandit Feedback (2024)0.00
• Learning In Restless Bandits Under Exogenous Global Markov Process (2021)6.34
• A Closer Look At The Worst-case Behavior Of Multi-armed Bandit Algorithms (2021)0.00
• Computing The Performance Of A New Adaptive Sampling Algorithm Based On The Gittins Index In Experiments With Exponential Rewards (2023)0.00
• Optimal Policies For Observing Time Series And Related Restless Bandit Problems (2017)0.00