1Deep Learning for Video Game PlayingNiels Justesen 1 , Philip Bontrager 2 , Julian Togelius 2 , Sebastian Risi 11 IT University of Copenhagen, Copenhagen2 New York University, New YorkIn this article, we review recent Deep Learning advances inthe context of how they have been applied to play different typesof video games such as f i rst-person shooters, arcade games, andreal-time strategy games. We analyze the unique requirementsthat different game genres pose to a deep learning system andhighlight important open challenges in the context of applyingthese machine learning methods to video games, such as generalgame playing, dealing with extremely large decision spaces andsparse rewards.I. I NTRODUCTIONApplying AI techniques to games is now an established re-search f ield with multiple conferences and dedicated journals.In this article, we review recent advances in deep learning forvideo game playing and employed game research platformswhile highlighting important open challenges. A main moti-vation for writing this article is to review the f ield from theperspective of different types of games, the challenges theypose for deep learning, and how deep learning can be usedto play these games. A variety of different review articles ondeep learning exists [31], [66], [107], as well as surveys onreinforcement learning [119] and deep reinforcement learn-ing [72], here we focus on these techniques applied to videogame playing.In particular, in this article, we focus on game problems andenvironments that have been used extensively for DL-basedGame AI, such as Atari/ALE, Doom, Minecraft, StarCraft andcar racing. Additionally, we review existing work and pointout important challenges that remain to be solved. We areinterested in approaches that aim to play a particular videogame well (in contrast to board games such as Go, etc.), frompixels or feature vectors, without an existing forward model.Several game genres are analyzed to point out the many anddiverse challenges they pose to human and machine players.It is important to note that there are many uses of AI inand for games that we are not covering in this article; AI andgames is a large and diverse f ield [144], [143], [78], [30],[83]. In this paper, we focus on deep learning methods forplaying video games well, but there is also plenty of researchfor playing games in a believable, entertaining or human-likemanner [47]. Furthermore, AI is commonly used for tasks thatdo not involve playing the game, such as modeling players’behavior, experience or preferences [142], or generating gamecontent such as levels, textures or rules [109]. Deep learningis also far from the only AI method that has applications ingames, other prominent methods include Monte Carlo TreeSearch [15] and evolutionary computation [98], [75]. In whatfollows, it is important to be aware of the limitations of thescope of this article.The paper is structured as follows: The next section gives anoverview of different deep learning methods applied to games,followed by the different research platforms that are currentlyin use. Section IV reviews the use of DL methods in differentvideo game types and Section V gives a historical overviewof the f ield. We conclude the paper by pointing out importantopen challenges in Section VI and a conclusion in Section VII.II. D EEP L EARNING O VERVIEWMachine learning is traditionally divided into three differenttypes of learning: supervised learning, unsupervised learning,and reinforcement learning. Additionally, one could also usestochastic optimization approaches like evolutionary computa-tion for learning. All of these are viable candidates for trainingdeep networks to play games. In this section, we give a briefoverview of these approaches, and also of hybrid approachesthat combine one or several of these or other methods together.A. Supervised LearningIn supervised training of artif icial neural networks (ANNs),an agent learns by example [65], [99]. During training, anagent is asked to make a decision for which the correct answeris already known. After the decision is made, an error functionis used to determine the difference between the providedanswer and the ground truth, which is used as a loss to updatethe model. The goal is to achieve a model that can generalizebeyond the training data and thus perform well on examples ithas never seen before, which usually require a large data set.The architectures of these neural networks can roughly bedivided into two major categories: feedforward and recurrentneural networks (RNN). Feedforward networks take a singleinput, for example, a representation of the game state, andselect an output from a predef ined set of possible outputs.Famously this is done with image classif ication, where animage is provided and a label pertaining to what is in theimage is output. Feedforward networks with convolution, orconvolutional neural networks (CNN), have been the mostsuccessful way to process images and this type of architectureis thus highly relevant when processing raw image data froma video game.In a CNN, some layers (also called convolutional layers)consist of a number of trainable f ilters [68]. These f ilters areconvolved across the output of the previous layer. At eachconvolution, the dot product is taken between the f ilter weightsand that section of the input values. These dot products arenormally passed through an activation function f irst. Thesevalues then make the next layer. Generally, these f ilters eachlearn to respond to certain features in the data, allowing thenetwork to eff iciently classify objects.RNNs are typically applied to time series data, in whichthe output of the network can dependent on the network’sarXiv:1708.07902v2 [cs.AI] 30 Oct 2017
