Preventing Imitation Learning With Adversarial Policy Ensembles

Abstract

Imitation learning can reproduce policies by observing experts, which poses a problem regarding policy privacy. Policies, such as human, or policies on deployed robots, can all be cloned without consent from the owners. How can we protect against external observers cloning our proprietary policies? To answer this question we introduce a new reinforcement learning framework, where we train an ensemble of near-optimal policies, whose demonstrations are guaranteed to be useless for an external observer. We formulate this idea by a constrained optimization problem, where the objective is to improve proprietary policies, and at the same time deteriorate the virtual policy of an eventual external observer. We design a tractable algorithm to solve this new optimization problem by modifying the standard policy gradient algorithm. Our formulation can be interpreted in lenses of confidentiality and adversarial behaviour, which enables a broader perspective of this work. We demonstrate the existe

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhan2020preventing

Related papers

• Imitating Opponent To Win: Adversarial Policy Imitation Learning In Two-player Competitive Games (2022)0.00
• Toward Evaluating Robustness Of Reinforcement Learning With Adversarial Policy (2023)4.52
• Adversarial Soft Advantage Fitting: Imitation Learning Without Policy Optimization (2020)0.00
• Epopt: Learning Robust Neural Network Policies Using Model Ensembles (2016)0.00
• Robust Opponent Modeling Via Adversarial Ensemble Reinforcement Learning In Asymmetric Imperfect-information Games (2019)0.00
• Adversarial Policies: Attacking Deep Reinforcement Learning (2019)0.00
• Online Robust Policy Learning In The Presence Of Unknown Adversaries (2018)0.00
• Online Adaptation For Enhancing Imitation Learning Policies (2024)0.00