188宝金博页面版

Implementing the Deep Q-NetworkMelrose RoderickHumans To Robots LaboratoryBrown UniversityProvidence, RI 02912melrose_roderick@brown.eduJames MacGlashanHumans To Robots LaboratoryBrown UniversityProvidence, RI 02912jmacglashan@gmail.comStefanie TellexHumans To Robots LaboratoryBrown UniversityProvidence, RI 02912stefie10@cs.brown.eduAbstractThe Deep Q-Network proposed by Mnih et al. [2015] has become a benchmarkand building point for much deep reinforcement learning research. However,replicating results for complex systems is often challenging since original scientif i cpublications are not always able to describe in detail every important parametersetting and software engineering solution. In this paper, we present results fromour work reproducing the results of the DQN paper. We highlight key areas inthe implementation that were not covered in great detail in the original paperto make it easier for researchers to replicate these results, including terminationconditions and gradient descent algorithms. Finally, we discuss methods forimproving the computational performance and provide our own implementationthat is designed to work with a range of domains, and not just the original ArcadeLearning Environment [Bellemare et al., 2013].1 IntroductionOver the past few years, deep reinforcement learning has gained much popularity as it has been shownto perform better than previous methods on domains with very large state-spaces. In one of the earliestdeep reinforcement learning papers (hereafter the DQN paper), Mnih et al. [2015] presented a methodfor learning to play Atari 2600 video games, using the Arcade Learning Environment (ALE) [Belle-mare et al., 2013], from image and performance data alone using the same deep neural networkarchitecture and hyper-parameters for all the games. DQN outperformed previous reinforcementlearning methods on nearly all of the games and recorded better than human performance on most. Asmany researchers tackle reinforcement learning problems with deep reinforcement learning methodsand propose alternative algorithms, the results of the DQN paper are often used as a benchmark toshow improvement. Thus, implementing the DQN algorithm is important for both replicating theresults of the DQN paper for comparison and also building off the original algorithm. One of themain contributions of the DQN paper was f i nding ways to improve stability in their artif i cial neuralnetworks during training. There are, however, a number of other areas in the implementation of thismethod that are crucial to its success, which were only mentioned brief l y in the paper.We implemented a Deep Q-Network (DQN) to play the Atari games and replicated the results ofMnih et al. [2015]. Our implementation, available freely online, 1 runs around 4x faster than the1 www.github.com/h2r/burlap_caffe30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.arXiv:1711.07478v1 [cs.LG] 20 Nov 2017