Deep Ordinal Reinforcement Learning

Abstract

Reinforcement learning usually makes use of numerical rewards, which have nice properties but also come with drawbacks and difficulties. Using rewards on an ordinal scale (ordinal rewards) is an alternative to numerical rewards that has received more attention in recent years. In this paper, a general approach to adapting reinforcement learning problems to the use of ordinal rewards is presented and motivated. We show how to convert common reinforcement learning algorithms to an ordinal variation by the example of Q-learning and introduce Ordinal Deep Q-Networks, which adapt deep reinforcement learning to ordinal rewards. Additionally, we run evaluations on problems provided by the OpenAI Gym framework, showing that our ordinal variants exhibit a performance that is comparable to the numerical variations for a number of problems. We also give first evidence that our ordinal variant is able to produce better results for problems with less engineered and simpler-to-design reward signals.

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keyzap2019deep

Related papers

• Quantile-based Deep Reinforcement Learning Using Two-timescale Policy Gradient Algorithms (2023)0.00
• An Information-theoretic Optimality Principle For Deep Reinforcement Learning (2017)0.00
• Elo-rated Sequence Rewards: Advancing Reinforcement Learning Models (2024)0.00
• Deep Reinforcement Learning From Hierarchical Preference Design (2023)2.00
• Average Reward Reinforcement Learning For Omega-regular And Mean-payoff Objectives (2025)0.00
• Self Punishment And Reward Backfill For Deep Q-learning (2020)7.16
• Distributional Reinforcement Learning For Multi-dimensional Reward Functions (2021)0.00
• Adaptive Symmetric Reward Noising For Reinforcement Learning (2019)0.00