Directed Exploration In Reinforcement Learning From Linear Temporal Logic

Abstract

Linear temporal logic (LTL) is a powerful language for task specification in reinforcement learning, as it allows describing objectives beyond the expressivity of conventional discounted return formulations. Nonetheless, recent works have shown that LTL formulas can be translated into a variable rewarding and discounting scheme, whose optimization produces a policy maximizing a lower bound on the probability of formula satisfaction. However, the synthesized reward signal remains fundamentally sparse, making exploration challenging. We aim to overcome this limitation, which can prevent current algorithms from scaling beyond low-dimensional, short-horizon problems. We show how better exploration can be achieved by further leveraging the LTL specification and casting its corresponding Limit Deterministic B\"uchi Automaton (LDBA) as a Markov reward process, thus enabling a form of high-level value estimation. By taking a Bayesian perspective over LDBA dynamics and proposing a suitable prio

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybagatella2024directed

Related papers

• Sample-efficient Reinforcement Learning With Temporal Logic Objectives: Leveraging The Task Specification To Guide Exploration (2024)0.00
• A Policy Search Method For Temporal Logic Specified Reinforcement Learning Tasks (2017)11.58
• Adaptive Reward Design For Reinforcement Learning (2024)0.00
• Sample Efficient Model-free Reinforcement Learning From LTL Specifications With Optimality Guarantees (2023)0.00
• Logical Specifications-guided Dynamic Task Sampling For Reinforcement Learning Agents (2024)2.26
• Learning Probabilistic Temporal Logic Specifications For Stochastic Systems (2025)0.00
• Funnel-based Reward Shaping For Signal Temporal Logic Tasks In Reinforcement Learning (2022)7.16
• On The (in)tractability Of Reinforcement Learning For LTL Objectives (2021)0.00