Estimating Scale-invariant Future In Continuous Time

Abstract

Natural learners must compute an estimate of future outcomes that follow from a stimulus in continuous time. Widely used reinforcement learning algorithms discretize continuous time and estimate either transition functions from one step to the next (model-based algorithms) or a scalar value of exponentially-discounted future reward using the Bellman equation (model-free algorithms). An important drawback of model-based algorithms is that computational cost grows linearly with the amount of time to be simulated. On the other hand, an important drawback of model-free algorithms is the need to select a time-scale required for exponential discounting. We present a computational mechanism, developed based on work in psychology and neuroscience, for computing a scale-invariant timeline of future outcomes. This mechanism efficiently computes an estimate of inputs as a function of future time on a logarithmically-compressed scale, and can be used to generate a scale-invariant power-law-discoun

Stats

• citations18
• S2 citations—
• github stars0
• HF likes0
• heat score9.59
• arxiv keytiganj2018estimating

Related papers

• Deep Reinforcement Learning With Time-scale Invariant Memory (2024)0.00
• Learning Dynamics Model In Reinforcement Learning By Incorporating The Long Term Future (2019)0.00
• Efficient Exploration In Continuous-time Model-based Reinforcement Learning (2023)0.00
• Scale-invariant Temporal History (SITH): Optimal Slicing Of The Past In An Uncertain World (2017)0.00
• Learning Successor States And Goal-dependent Values: A Mathematical Viewpoint (2021)0.00
• Learning When To Act: Interval-aware Reinforcement Learning With Predictive Temporal Structure (2026)0.00
• An Idiosyncrasy Of Time-discretization In Reinforcement Learning (2024)0.00
• An Online Prediction Algorithm For Reinforcement Learning With Linear Function Approximation Using Cross Entropy Method (2018)7.16