Understanding Behavioral Metric Learning: A Large-scale Study On Distracting Reinforcement Learning Environments

Abstract

A key approach to state abstraction is approximating behavioral metrics (notably, bisimulation metrics) in the observation space and embedding these learned distances in the representation space. While promising for robustness to task-irrelevant noise, as shown in prior work, accurately estimating these metrics remains challenging, requiring various design choices that create gaps between theory and practice. Prior evaluations focus mainly on final returns, leaving the quality of learned metrics and the source of performance gains unclear. To systematically assess how metric learning works in deep reinforcement learning (RL), we evaluate five recent approaches, unified conceptually as isometric embeddings with varying design choices. We benchmark them with baselines across 20 state-based and 14 pixel-based tasks, spanning 370 task configurations with diverse noise settings. Beyond final returns, we introduce the evaluation of a denoising factor to quantify the encoder's ability to filt

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyluo2025understanding

Related papers

• Towards Robust Bisimulation Metric Learning (2021)0.00
• Goal-conditioned Reinforcement Learning From Sub-optimal Data On Metric Spaces (2024)0.00
• FOCAL: Efficient Fully-offline Meta-reinforcement Learning Via Distance Metric Learning And Behavior Regularization (2020)0.00
• Metrics And Continuity In Reinforcement Learning (2021)0.00
• An Analysis Of Model-based Reinforcement Learning From Abstracted Observations (2022)0.00
• The Loca Regret: A Consistent Metric To Evaluate Model-based Behavior In Reinforcement Learning (2020)0.00
• S-RL Toolbox: Environments, Datasets And Evaluation Metrics For State Representation Learning (2018)0.00
• Quantifying First-order Markov Violations In Noisy Reinforcement Learning: A Causal Discovery Approach (2025)0.00