Predecessor Features

Abstract

Any reinforcement learning system must be able to identify which past events contributed to observed outcomes, a problem known as credit assignment. A common solution to this problem is to use an eligibility trace to assign credit to recency-weighted set of experienced events. However, in many realistic tasks, the set of recently experienced events are only one of the many possible action events that could have preceded the current outcome. This suggests that reinforcement learning can be made more efficient by allowing credit assignment to any viable preceding state, rather than only those most recently experienced. Accordingly, we examine ``Predecessor Features'', the fully bootstrapped version of van Hasselt's ``Expected Trace'', an algorithm that achieves this richer form of credit assignment. By maintaining a representation that approximates the expected sum of past occupancies, this algorithm allows temporal difference (TD) errors to be propagated accurately to a larger number of

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybailey2022predecessor

Related papers

• Expected Eligibility Traces (2020)0.00
• Recall Traces: Backtracking Models For Efficient Reinforcement Learning (2018)0.00
• Trajectory-aware Eligibility Traces For Off-policy Reinforcement Learning (2023)0.00
• Advantages And Limitations Of Using Successor Features For Transfer In Reinforcement Learning (2017)0.00
• Demystifying The Recency Heuristic In Temporal-difference Learning (2024)0.00
• Successor Features Combine Elements Of Model-free And Model-based Reinforcement Learning (2019)0.00
• Transfer With Model Features In Reinforcement Learning (2018)0.00
• Improving Experience Replay With Successor Representation (2021)0.00