Meta-reinforcement Learning For The Tuning Of PI Controllers: An Offline Approach

Abstract

Meta-learning is a branch of machine learning which trains neural network models to synthesize a wide variety of data in order to rapidly solve new problems. In process control, many systems have similar and well-understood dynamics, which suggests it is feasible to create a generalizable controller through meta-learning. In this work, we formulate a meta reinforcement learning (meta-RL) control strategy that can be used to tune proportional--integral controllers. Our meta-RL agent has a recurrent structure that accumulates "context" to learn a system's dynamics through a hidden state variable in closed-loop. This architecture enables the agent to automatically adapt to changes in the process dynamics. In tests reported here, the meta-RL agent was trained entirely offline on first order plus time delay systems, and produced excellent results on novel systems drawn from the same distribution of process dynamics used for training. A key design element is the ability to leverage model-bas

Stats

• citations39
• S2 citations—
• github stars0
• HF likes0
• heat score12.02
• arxiv keymcclement2022meta

Related papers

• Towards An Adaptable And Generalizable Optimization Engine In Decision And Control: A Meta Reinforcement Learning Approach (2024)0.00
• Metatrace Actor-critic: Online Step-size Tuning By Meta-gradient Descent For Reinforcement Learning Control (2018)0.00
• Offline Meta-reinforcement Learning With Online Self-supervision (2021)0.00
• Model Predictive Control And Reinforcement Learning: A Unified Framework Based On Dynamic Programming (2024)10.61
• Pid-inspired Inductive Biases For Deep Reinforcement Learning In Partially Observable Control Tasks (2023)0.00
• Leveraging Offline Data In Online Reinforcement Learning (2022)0.00
• Finetuning From Offline Reinforcement Learning: Challenges, Trade-offs And Practical Solutions (2023)0.00
• Offline Meta-reinforcement Learning With Advantage Weighting (2020)0.00