Implicit Language Model in LSTM for OCREkraam SabirInformation Sciences InstituteUniversity of Southern CaliforniaMarina Del Rey, California 90007Email: esabir@isi.eduStephen RawlsInformation Sciences InstituteUniversity of Southern CaliforniaMarina Del Rey, California 90007Email:srawls@isi.eduPrem NatarajanInformation Sciences InstituteUniversity of Southern CaliforniaMarina Del Rey, California 90007Email: pnataraj@isi.eduAbstract—Neural networks have become the technique ofchoice for OCR, but many aspects of how and why they deliversuperior performance are still unknown. One key differencebetween current neural network techniques using LSTMs andthe previous state-of-the-art HMM systems is that HMM systemshave a strong independence assumption. In comparison LSTMshave no explicit constraints on the amount of context that canbe considered during decoding. In this paper we show that theylearn an implicit LM and attempt to characterize the strengthof the LM in terms of equivalent n-gram context. We show thatthis implicitly learned language model provides a 2.4% CERimprovement on our synthetic test set when compared againsta test set of random characters (i.e. not naturally occurringsequences), and that the LSTM learns to use up to 5 charactersof context (which is roughly 88 frames in our conf i guration). Webelieve that this is the f i rst ever attempt at characterizing thestrength of the implicit LM in LSTM based OCR systems.I. I NTRODUCTIONAt the heart of any Optical Character Recognition (OCR)system is a glyph recognition model whose purpose is to iden-tify individual glyphs based on extracted features. However, inaddition to the glyph model, OCR systems typically employfeature extraction, segmentation and language modeling mod-ules to get competitive performance [1]. Of these, languagemodeling improves the output of a glyph recognition modelconditioned on the distribution of characters or words fromtask-relevant-text i.e. a language model. A language modelcan build a word or a character language model [2]. [3] givesa general survey on the use of language modeling. Irrespectiveof the OCR method, language modeling has been investigatedindependently and has also played a crucial role in achievingbetter performance [4][5].Hidden Markov Model (HMM) based systems providedsegmentation free OCR and outperformed then existing seg-mentation based approaches [6][7]. This conditional indepen-dence limitation was addressed by Recurrent Neural Networks(RNNs) [8] which theoretically have no limitations on thelength of context they can utilize.While Neural Networks have been used with success forOCR in the past [9][10], it is only recently that recurrentnetworks and particularly LSTMs [11] became popular forthe OCR task, improving upon the performance of HMMOCR systems [12][13]. Solutions to the vanishing and ex-ploding gradient problems associated with the training ofRNNs [14][11] coupled with the introduction of ConnectionistTemporal Classif i cation (CTC) loss [15] played a major role inthis resurgence. The CTC loss was particularly well suited totackling the OCR problem, removing the necessity for framelevel label assignment.The convincing performance improvements made byLSTMs however stand in stark contrast to the limited inter-pretability of these networks. The functionality of individualneurons, weights and to some extent the hidden layers them-selves remains ambiguous at best. To this end, a signif i cantamount of effort has been expended in explaining LSTMsregarding their structure such as performance of LSTM vari-ants with and without hyperparameter tuning and the effectsof depth [16][17][18]. [19] and [20] explore the memoryand functionality of neurons with experiments that test longterm reasoning among others, on a character language modellearning task.Continuing in the general direction of unraveling LSTMs,we explore their possibility of learning a language model whentrained on a different but related OCR task. Foundationalcredibility for LSTMs learning an internal language modelwhen trained for OCR can be enumerated from previousdiscussion as follows: 1) LSTMs do not have an explicitrestriction on the amount of context they can learn; 2) Theyhave been shown to learn character language models whentrained for it specif i cally as in [20]; and 3) Learning a languagemodel in general helps improve performance on the OCRtask. We f i nd additional evidence for this idea in [21] wherean LSTM is trained on a multilingual OCR task. The setupinvolves testing multiple LSTM models which are trained onone native language and tested on other foreign languages withthe same glyphs. The results on a real world problem showup to 3.6% CER difference in performance when testing onforeign languages, which is indicative of the model’s relianceon the native language model. However, the authors of [21]do not explain this phenomena.In this paper we attempt to advance our scientif i c un-derstanding of LSTMs, particularly the interactions betweenlanguage model and glyph model present within an LSTM.We call this internal language model the implicit languagemodel (implicit LM). Our contributions in this paper include:1) Establishing the presence of implicit LM under controlledconditions; and 2) characterizing the nature of implicit LMby f i nding how many characters of context it makes useof. The implicit LM we characterize is different from thelanguage model in [19][20] discussed above in that the settingarXiv:1805.09441v1 [cs.CV] 23 May 2018
