Deep Reinforcement Learning From Hierarchical Preference Design

Abstract

Reward design is a fundamental, yet challenging aspect of reinforcement learning (RL). Researchers typically utilize feedback signals from the environment to handcraft a reward function, but this process is not always effective due to the varying scale and intricate dependencies of the feedback signals. This paper shows by exploiting certain structures, one can ease the reward design process. Specifically, we propose a hierarchical reward modeling framework -- HERON for scenarios: (I) The feedback signals naturally present hierarchy; (II) The reward is sparse, but with less important surrogate feedback to help policy learning. Both scenarios allow us to design a hierarchical decision tree induced by the importance ranking of the feedback signals to compare RL trajectories. With such preference data, we can then train a reward model for policy learning. We apply HERON to several RL applications, and we find that our framework can not only train high performing agents on a variety of dif

Stats

• citations0
• S2 citations—
• github stars9
• HF likes0
• heat score2.00
• arxiv keybukharin2023deep

Related papers

• Reinforcement Learning From Diverse Human Preferences (2023)0.00
• Reward Design For Reinforcement Learning Agents (2025)0.00
• MENTOR: Guiding Hierarchical Reinforcement Learning With Human Feedback And Dynamic Distance Constraint (2024)6.34
• Tiered Reward: Designing Rewards For Specification And Fast Learning Of Desired Behavior (2022)0.00
• Hierarchical Reinforcement Learning With Advantage-based Auxiliary Rewards (2019)0.00
• Learning Representations In Model-free Hierarchical Reinforcement Learning (2018)11.49
• Reward Models In Deep Reinforcement Learning: A Survey (2025)0.00
• Hierarchical Reinforcement Learning Via Advantage-weighted Information Maximization (2019)0.00