Exploiting Hierarchy For Learning And Transfer In Kl-regularized RL

Abstract

As reinforcement learning agents are tasked with solving more challenging and diverse tasks, the ability to incorporate prior knowledge into the learning system and to exploit reusable structure in solution space is likely to become increasingly important. The KL-regularized expected reward objective constitutes one possible tool to this end. It introduces an additional component, a default or prior behavior, which can be learned alongside the policy and as such partially transforms the reinforcement learning problem into one of behavior modelling. In this work we consider the implications of this framework in cases where both the policy and default behavior are augmented with latent variables. We discuss how the resulting hierarchical structures can be used to implement different inductive biases and how their modularity can benefit transfer. Empirically we find that they can lead to faster learning and transfer on a range of continuous control tasks.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytirumala2019exploiting

Related papers

• Information Asymmetry In Kl-regularized RL (2019)0.00
• Multi-horizon Representations With Hierarchical Forward Models For Reinforcement Learning (2022)0.00
• Learning Representations In Model-free Hierarchical Reinforcement Learning (2018)11.49
• A Kl-regularization Framework For Learning To Plan With Adaptive Priors (2025)0.00
• Deep Reinforcement Learning From Hierarchical Preference Design (2023)2.00
• Rethinking KL Regularization In RLHF: From Value Estimation To Gradient Optimization (2025)0.00
• Boosting Hierarchical Reinforcement Learning With Meta-learning For Complex Task Adaptation (2024)0.00
• Toward Robust Long Range Policy Transfer (2021)0.00