Online Reinforcement Learning Via Sparse Gaussian Mixture Model Q-functions

Abstract

This paper introduces a structured and interpretable online policy-iteration framework for reinforcement learning (RL), built around the novel class of sparse Gaussian mixture model Q-functions (S-GMM-QFs). Extending earlier work that trained GMM-QFs offline, the proposed framework develops an online scheme that leverages streaming data to encourage exploration. Model complexity is regulated through sparsification by Hadamard overparametrization, which mitigates overfitting while preserving expressiveness. The parameter space of S-GMM-QFs is naturally endowed with a Riemannian manifold structure, allowing for principled parameter updates via online gradient descent on a smooth objective. Numerical experiments show that S-GMM-QFs match or even outperform dense deep RL (DeepRL) methods on standard benchmarks while using significantly fewer parameters. Moreover, they maintain strong performance even in low-parameter regimes where sparsified DeepRL methods fail to generalize.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvu2025online

Related papers

• Evolving Diffusion And Flow Matching Policies For Online Reinforcement Learning (2025)0.00
• Fast Reinforcement Learning With Incremental Gaussian Mixture Models (2020)0.00
• Boosting Offline Reinforcement Learning With Residual Generative Modeling (2021)0.00
• Sample Efficient Active Algorithms For Offline Reinforcement Learning (2026)0.00
• Task-agnostic Online Reinforcement Learning With An Infinite Mixture Of Gaussian Processes (2020)0.00
• Entropy-regularized Diffusion Policy With Q-ensembles For Offline Reinforcement Learning (2024)3.58
• Uncertainty-based Offline Reinforcement Learning With Diversified Q-ensemble (2021)0.00
• Probabilistic Mixture-of-experts For Efficient Deep Reinforcement Learning (2021)0.00