Computationally Efficient Reinforcement Learning: Targeted Exploration Leveraging Simple Rules

Abstract

Model-free Reinforcement Learning (RL) generally suffers from poor sample complexity, mostly due to the need to exhaustively explore the state-action space to find well-performing policies. On the other hand, we postulate that expert knowledge of the system often allows us to design simple rules we expect good policies to follow at all times. In this work, we hence propose a simple yet effective modification of continuous actor-critic frameworks to incorporate such rules and avoid regions of the state-action space that are known to be suboptimal, thereby significantly accelerating the convergence of RL agents. Concretely, we saturate the actions chosen by the agent if they do not comply with our intuition and, critically, modify the gradient update step of the policy to ensure the learning process is not affected by the saturation step. On a room temperature control case study, it allows agents to converge to well-performing policies up to 6-7x faster than classical agents without comp

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keydinatale2022computationally

Related papers

• Efficient Reinforcement Learning Via Decoupling Exploration And Utilization (2023)2.56
• Decoupled Exploration And Exploitation Policies For Sample-efficient Reinforcement Learning (2021)0.00
• Conservative Exploration For Policy Optimization Via Off-policy Policy Evaluation (2023)0.00
• Human-inspired Framework To Accelerate Reinforcement Learning (2023)0.00
• Simple Agent, Complex Environment: Efficient Reinforcement Learning With Agent States (2021)0.00
• PC-MLP: Model-based Reinforcement Learning With Policy Cover Guided Exploration (2021)0.00
• Random Latent Exploration For Deep Reinforcement Learning (2024)0.00
• Fast Exploration With Simplified Models And Approximately Optimistic Planning In Model Based Reinforcement Learning (2018)0.00