arXiv:1708.02182v1 [cs.CL] 7 Aug 2017Regularizing and Optimizing LSTM Language ModelsStephen Merity 1 Nitish Shirish Keskar 1 Richard Socher 1AbstractRecurrent neural networks (RNNs), such as longshort-term memory networks (LSTMs), serve asa fundamental building block for many sequencelearning tasks, including machine translation,language modeling, and question answering. Inthis paper, we consider the specif i c problem ofword-level language modeling and investigatestrategiesforregularizingandoptimizingLSTM-based models. We propose the weight-droppedLSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regulariza-tion. Further, we introduce NT-ASGD, a vari-ant of the averaged stochastic gradient method,wherein the averaging trigger is determined us-ing a non-monotoniccondition as opposed to be-ing tuned by the user. Using these and other reg-ularization strategies, we achieve state-of-the-artword level perplexities on two data sets: 57.3 onPenn Treebank and 65.8 on WikiText-2. In ex-ploringtheeffectivenessofaneuralcacheincon-junction with our proposedmodel, we achieve aneven lower state-of-the-art perplexity of 52.8 onPenn Treebank and 52.0 on WikiText-2.1. IntroductionEffective regularization techniques for deep learning havebeen the subject of much research in recent years. Giventhe over-parameterization of neural networks, general-ization performance crucially relies on the ability toregularize the models suff i ciently. Strategies such asdropout (Srivastava et al., 2014) and batch normalization(Ioffe & Szegedy, 2015) have found great success and arenow ubiquitous in feed-forward and convolutional neuralnetworks. Naïvely applying these approaches to the caseof recurrent neural networks (RNNs) has not been highlysuccessful however. Many recent works have hence beenfocused on the extension of these regularization strategiesto RNNs; we brief l y discuss some of them below.1 Salesforce Research, Palo Alto, USA. Correspondence to:Stephen Merity <smerity@salesforce.com>.A naïve application of dropout (Srivastava et al., 2014)to an RNN’s hidden state is ineffective as it disruptsthe RNN’s ability to retain long term dependencies(Zaremba et al., 2014). Gal & Ghahramani (2016) proposeovercoming this problem by retaining the same dropoutmask across multiple time steps as opposed to samplinga new binary mask at each timestep. Another approachis to regularize the network through limiting updates tothe RNN’s hidden state. One such approach is taken bySemeniuta et al. (2016) wherein the authors drop updatesto network units, specif i cally the input gates of the LSTM,in lieu of the units themselves. This is reminiscent of zone-out (Krueger et al., 2016) where updates to the hidden statemay fail to occur for randomly selected neurons.Instead of operating on the RNN’s hidden states, one canregularize the network through restrictions on the recur-rent matrices as well. This can be done either throughrestricting the capacity of the matrix (Arjovsky et al.,2016; Wisdom et al., 2016; Jing et al., 2016) or throughelement-wise interactions (Balduzzi & Ghifary, 2016;Bradbury et al., 2016; Seo et al., 2016).Other forms of regularization explicitly act upon activa-tions such as batch normalization (Ioffe & Szegedy, 2015),recurrentbatchnormalization(Cooijmans et al.,2016), andlayer normalization (Ba et al., 2016). These all introduceadditionaltrainingparametersandcancomplicatethetrain-ing process while increasing the sensitivity of the model.Inthiswork,weinvestigateaset ofregularizationstrategiesthatare notonlyhighlyeffectivebut whichcanalso beusedwith no modif i cation to existing LSTM implementations.The weight-dropped LSTM applies recurrent regulariza-tion through a DropConnect mask on the hidden-to-hiddenrecurrent weights. Other strategies include the use ofrandomized-lengthbackpropagation through time (BPTT),embedding dropout, activation regularization (AR), andtemporal activation regularization (TAR).As no modif i cations are required of the LSTM implemen-tation these regularization strategies are compatible withblack box libraries, such as NVIDIA cuDNN, which canbe many times faster than naïve LSTM implementations.Effective methods for training deep recurrent networkshave also been a topic of renewed interest. Once a model
