Approximate Information State For Approximate Planning And Reinforcement Learning In Partially Observed Systems

Abstract

We propose a theoretical framework for approximate planning and learning in partially observed systems. Our framework is based on the fundamental notion of information state. We provide two equivalent definitions of information state -- i) a function of history which is sufficient to compute the expected reward and predict its next value; ii) equivalently, a function of the history which can be recursively updated and is sufficient to compute the expected reward and predict the next observation. An information state always leads to a dynamic programming decomposition. Our key result is to show that if a function of the history (called approximate information state (AIS)) approximately satisfies the properties of the information state, then there is a corresponding approximate dynamic program. We show that the policy computed using this is approximately optimal with bounded loss of optimality. We show that several approximations in state, observation and action spaces in literature can

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysubramanian2020approximate

Related papers

• Approximate Information State Based Convergence Analysis Of Recurrent Q-learning (2023)0.00
• Provably Efficient Reinforcement Learning In Partially Observable Dynamical Systems (2022)0.00
• Information-theoretic Methods For Planning And Learning In Partially Observable Markov Decision Processes (2016)0.00
• Learning Causal State Representations Of Partially Observable Environments (2019)0.00
• Provable Representation With Efficient Planning For Partial Observable Reinforcement Learning (2023)0.00
• Common Information Based Approximate State Representations In Multi-agent Reinforcement Learning (2021)0.00
• Reinforcement Learning From Partial Observation: Linear Function Approximation With Provable Sample Efficiency (2022)0.00
• Adaptive Approximate Policy Iteration (2020)0.00