Orchestrated Value Mapping For Reinforcement Learning

Abstract

We present a general convergent class of reinforcement learning algorithms that is founded on two distinct principles: (1) mapping value estimates to a different space using arbitrary functions from a broad class, and (2) linearly decomposing the reward signal into multiple channels. The first principle enables incorporating specific properties into the value estimator that can enhance learning. The second principle, on the other hand, allows for the value function to be represented as a composition of multiple utility functions. This can be leveraged for various purposes, e.g. dealing with highly varying reward scales, incorporating a priori knowledge about the sources of reward, and ensemble learning. Combining the two principles yields a general blueprint for instantiating convergent algorithms by orchestrating diverse mapping functions over multiple reward channels. This blueprint generalizes and subsumes algorithms such as Q-Learning, Log Q-Learning, and Q-Decomposition. In additi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfatemi2022orchestrated

Related papers

• The Value-improvement Path: Towards Better Representations For Reinforcement Learning (2020)6.77
• Convex Programs And Lyapunov Functions For Reinforcement Learning: A Unified Perspective On The Analysis Of Value-based Methods (2022)2.26
• Greedy-based Value Representation For Optimal Coordination In Multi-agent Reinforcement Learning (2021)0.00
• Greedy Based Value Representation For Optimal Coordination In Multi-agent Reinforcement Learning (2022)0.00
• Value Function Decomposition For Iterative Design Of Reinforcement Learning Agents (2022)0.00
• Stackelberg Coupling Of Online Representation Learning And Reinforcement Learning (2025)0.00
• Distributional Reinforcement Learning For Multi-dimensional Reward Functions (2021)0.00
• Uneven: Universal Value Exploration For Multi-agent Reinforcement Learning (2020)0.00