Adapting The Function Approximation Architecture In Online Reinforcement Learning

Abstract

The performance of a reinforcement learning (RL) system depends on the computational architecture used to approximate a value function. Deep learning methods provide both optimization techniques and architectures for approximating nonlinear functions from noisy, high-dimensional observations. However, prevailing optimization techniques are not designed for strictly-incremental online updates. Nor are standard architectures designed for observations with an a priori unknown structure: for example, light sensors randomly dispersed in space. This paper proposes an online RL prediction algorithm with an adaptive architecture that efficiently finds useful nonlinear features. The algorithm is evaluated in a spatial domain with high-dimensional, stochastic observations. The algorithm outperforms non-adaptive baseline architectures and approaches the performance of an architecture given side-channel information. These results are a step towards scalable RL algorithms for more general problems,

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymartin2021adapting

Related papers

• An Online Prediction Algorithm For Reinforcement Learning With Linear Function Approximation Using Cross Entropy Method (2018)7.16
• Distributionally Robust Offline Reinforcement Learning With Linear Function Approximation (2022)0.00
• Online Sub-sampling For Reinforcement Learning With General Function Approximation (2021)0.00
• Provably Efficient Reinforcement Learning With Linear Function Approximation (2019)11.76
• Optimal Conservative Offline RL With General Function Approximation Via Augmented Lagrangian (2022)0.00
• Online Model Selection For Reinforcement Learning With Function Approximation (2020)0.00
• Improved Regret For Efficient Online Reinforcement Learning With Linear Function Approximation (2023)0.00
• Pessimistic Nonlinear Least-squares Value Iteration For Offline Reinforcement Learning (2023)0.00