Interpolating Between Softmax Policy Gradient And Neural Replicator Dynamics With Capped Implicit Exploration

Abstract

Neural replicator dynamics (NeuRD) is an alternative to the foundational softmax policy gradient (SPG) algorithm motivated by online learning and evolutionary game theory. The NeuRD expected update is designed to be nearly identical to that of SPG, however, we show that the Monte Carlo updates differ in a substantial way: the importance correction accounting for a sampled action is nullified in the SPG update, but not in the NeuRD update. Naturally, this causes the NeuRD update to have higher variance than its SPG counterpart. Building on implicit exploration algorithms in the adversarial bandit setting, we introduce capped implicit exploration (CIX) estimates that allow us to construct NeuRD-CIX, which interpolates between this aspect of NeuRD and SPG. We show how CIX estimates can be used in a black-box reduction to construct bandit algorithms with regret bounds that hold with high probability and the benefits this entails for NeuRD-CIX in sequential decision-making settings. Our ana

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymorrill2022interpolating

Related papers

• Neural Replicator Dynamics (2019)0.00
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00
• Exploring More When It Needs In Deep Reinforcement Learning (2021)0.00
• Exploration Conscious Reinforcement Learning Revisited (2018)0.00
• Nadpex: An On-policy Temporally Consistent Exploration Method For Deep Reinforcement Learning (2018)0.00
• An Alternate Policy Gradient Estimator For Softmax Policies (2021)0.00
• Learning Deterministic Policies With Policy Gradients In Constrained Markov Decision Processes (2025)0.00
• Improved Exploration Through Latent Trajectory Optimization In Deep Deterministic Policy Gradient (2019)0.00