Generalizing Multi-step Inverse Models For Representation Learning To Finite-memory Pomdps

Abstract

Discovering an informative, or agent-centric, state representation that encodes only the relevant information while discarding the irrelevant is a key challenge towards scaling reinforcement learning algorithms and efficiently applying them to downstream tasks. Prior works studied this problem in high-dimensional Markovian environments, when the current observation may be a complex object but is sufficient to decode the informative state. In this work, we consider the problem of discovering the agent-centric state in the more challenging high-dimensional non-Markovian setting, when the state can be decoded from a sequence of past observations. We establish that generalized inverse models can be adapted for learning agent-centric state representation for this task. Our results include asymptotic theory in the deterministic dynamics setting as well as counter-examples for alternative intuitive algorithms. We complement these findings with a thorough empirical study on the agent-centric s

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywu2024generalizing

Related papers

• Common Information Based Approximate State Representations In Multi-agent Reinforcement Learning (2021)0.00
• Provable RL With Exogenous Distractors Via Multistep Inverse Dynamics (2021)0.00
• Posterior Sampling-based Online Learning For Episodic Pomdps (2023)0.00
• Sample-efficient Learning Of Pomdps With Multiple Observations In Hindsight (2023)0.00
• Multistep Inverse Is Not All You Need (2024)1.20
• Low-dimensional State And Action Representation Learning With MDP Homomorphism Metrics (2021)0.00
• Learning Causal State Representations Of Partially Observable Environments (2019)0.00
• Scaling Internal-state Policy-gradient Methods For Pomdps (2025)0.00