Improved Exploration Through Latent Trajectory Optimization In Deep Deterministic Policy Gradient

Abstract

Model-free reinforcement learning algorithms such as Deep Deterministic Policy Gradient (DDPG) often require additional exploration strategies, especially if the actor is of deterministic nature. This work evaluates the use of model-based trajectory optimization methods used for exploration in Deep Deterministic Policy Gradient when trained on a latent image embedding. In addition, an extension of DDPG is derived using a value function as critic, making use of a learned deep dynamics model to compute the policy gradient. This approach leads to a symbiotic relationship between the deep reinforcement learning algorithm and the latent trajectory optimizer. The trajectory optimizer benefits from the critic learned by the RL algorithm and the latter from the enhanced exploration generated by the planner. The developed methods are evaluated on two continuous control tasks, one in simulation and one in the real world. In particular, a Baxter robot is trained to perform an insertion task, whil

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyluck2019improved

Related papers

• Learning To Explore With Meta-policy Gradient (2018)0.00
• ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm For Sparse Reward Continuous Control (2024)0.00
• Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control In Computationally Complex Environments (2019)0.00
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00
• Policy Augmentation: An Exploration Strategy For Faster Convergence Of Deep Reinforcement Learning Algorithms (2021)2.26
• Deep Reinforcement Learning With Feedback-based Exploration (2019)5.84
• Learning Optimal Deterministic Policies With Stochastic Policy Gradients (2024)0.00
• Guided Exploration In Reinforcement Learning Via Monte Carlo Critic Optimization (2022)0.00