Deterministic Implementations For Reproducibility In Deep Reinforcement Learning

Abstract

While deep reinforcement learning (DRL) has led to numerous successes in recent years, reproducing these successes can be extremely challenging. One reproducibility challenge particularly relevant to DRL is nondeterminism in the training process, which can substantially affect the results. Motivated by this challenge, we study the positive impacts of deterministic implementations in eliminating nondeterminism in training. To do so, we consider the particular case of the deep Q-learning algorithm, for which we produce a deterministic implementation by identifying and controlling all sources of nondeterminism in the training process. One by one, we then allow individual sources of nondeterminism to affect our otherwise deterministic implementation, and measure the impact of each source on the variance in performance. We find that individual sources of nondeterminism can substantially impact the performance of agent, illustrating the benefits of deterministic implementations. In addition,

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keynagarajan2018deterministic

Related papers

• On The Mistaken Assumption Of Interchangeable Deep Reinforcement Learning Implementations (2025)0.00
• Replicability In Reinforcement Learning (2023)0.00
• Let's Play Again: Variability Of Deep Reinforcement Learning Agents In Atari Environments (2019)0.00
• Dopamine: A Research Framework For Deep Reinforcement Learning (2018)0.00
• Beyond Expected Return: Accounting For Policy Reproducibility When Evaluating Reinforcement Learning Algorithms (2023)3.58
• Cleanba: A Reproducible And Efficient Distributed Reinforcement Learning Platform (2023)4.19
• Reinforcement Learning For Online Testing Of Autonomous Driving Systems: A Replication And Extension Study (2024)4.52
• Reproducibility Of Benchmarked Deep Reinforcement Learning Tasks For Continuous Control (2017)0.00