Does DQN Really Learn? Exploring Adversarial Training Schemes In Pong

Abstract

In this work, we study two self-play training schemes, Chainer and Pool, and show they lead to improved agent performance in Atari Pong compared to a standard DQN agent -- trained against the built-in Atari opponent. To measure agent performance, we define a robustness metric that captures how difficult it is to learn a strategy that beats the agent's learned policy. Through playing past versions of themselves, Chainer and Pool are able to target weaknesses in their policies and improve their resistance to attack. Agents trained using these methods score well on our robustness metric and can easily defeat the standard DQN agent. We conclude by using linear probing to illuminate what internal structures the different agents develop to play the game. We show that training agents with Chainer or Pool leads to richer network activations with greater predictive power to estimate critical game-state features compared to the standard DQN agent.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhe2022does

Related papers

• A Comparative Study Of Deep Reinforcement Learning Models: DQN Vs PPO Vs A2C (2024)0.00
• Generalization And Regularization In DQN (2018)0.00
• A Deep Reinforcement Learning Approach For Finding Non-exploitable Strategies In Two-player Atari Games (2022)0.00
• Transforming Game Play: A Comparative Study Of DCQN And DTQN Architectures In Reinforcement Learning (2024)0.00
• A Human Mixed Strategy Approach To Deep Reinforcement Learning (2018)7.50
• Visual Transfer Between Atari Games Using Competitive Reinforcement Learning (2018)7.50
• Convergent And Efficient Deep Q Network Algorithm (2021)0.00
• Reward Learning From Human Preferences And Demonstrations In Atari (2018)0.00