Revisiting the Hierarchical Multiscale LSTM´Akos K´ ad´ arTilburg Universitya.kadar@uvt.nlMarc-Alexandre Cˆ ot´ eMicrosoft Research Montrealmacote@microsoft.comGrzegorz Chrupa laTilburg Universityg.chrupala@uvt.nlAfra AlishahiTilburg Universitya.alishahi@uvt.nlAbstractHierarchical Multiscale LSTM (Chung et al., 2016a) is a state-of-the-art language modelthat learns interpretable structure from character-level input. Such models can pro-vide fertile ground for (cognitive) computational linguistics studies. However, the highcomplexity of the architecture, training procedure and implementations might hinder itsapplicability. We provide a detailed reproduction and ablation study of the architecture,shedding light on some of the potential caveats of re-purposing complex deep-learningarchitectures. We further show that simplifying certain aspects of the architecture can infact improve its performance. We also investigate the linguistic units (segments) learnedby various levels of the model, and argue that their quality does not correlate with theoverall performance of the model on language modeling.1 IntroductionVerifying and reproducing claims published in scientif i c articles is an essential part of building asolid foundation for future research. As such, reproduction has a long history in many scientif i cf i elds (Willett et al., 1985; Venables et al., 1993; Waltemath et al., 2011). Recent large-scalestudies, however, raise concerns about the reproducibility in a variety of areas and the potentialef f ect of this crisis (Baker, 2016). A reproduction study by the Open Science Collaboration (2015)estimates that only 40% of research in psychology is reproducible, while Begley and Ellis (2012)end up conf i rming only 11% of preclinical cancer studies. The latter work makes an importantlink between the low reproducibility rates and the notoriously low impact of preclinical cancerresearch on clinical practice (Hutchinson and Kirk, 2011).Our work is motivated by a similar concern, specif i cally the applicability of complex deeplearning architectures for computational (cognitive) linguistics research. State of the art sys-tems often employ complex architectures which integrate various design features and use manyoptimization techniques. Because the focus is on boosting the f i nal performance on a given task,often little ef f ort is put into understanding where the power of the system comes from. Thismakes these models much harder to adapt for new tasks or domains. In our view, it is essentialnot only to be able to reproduce reported results, but also to understand the contribution ofvarious design features through systematic ablation experiments.The higher performance brought by modern neural network architectures often comes at thecost of our understanding of the representations and structural information the system learns.However, for models to be generalizable to new domains, it is important to move towards ana-lyzing such structural representations, and investigating their impact on the f i nal performanceof the model.In the current study, we examine the reproducibility of a language model with the abilityto learn explicit linguistic structure: the Hierarchical Multiscale Recurrent Neural Network(HMLSTM) model. This architecture was introduced by Chung et al. (2016a) and set a newstate of the art on language-modeling benchmarks Text8, Hutter Prize and character-level PennTreebank. Additionally, the paper features examples where the lowest layer of the model recoversarXiv:1807.03595v1 [cs.CL] 10 Jul 2018
