188宝金博页面版

DeepThin: A Self-Compressing Library forDeep Neural NetworksMatthew Sotoudeh ∗Intel/UC Davismasotoudeh@ucdavis.eduSara S. BaghsorkhiIntelsara.s.baghsorkhi@intel.comAbstractAs the industry deploys increasingly large and complexneural networks to mobile devices, more pressure isput on the memory and compute resources of those de-vices.Deepcompression,orcompressionofdeepneuralnetwork weight matrices, is a technique to stretch re-sources for such scenarios. Existing compression meth-ods cannot ef f ectively compress models smaller than1-2% of their original size. We develop a new compres-siontechnique,DeepThin,buildingonexistingresearchin the area of low rank factorization. We identify andbreak artif i cial constraints imposed by low rank ap-proximations by combining rank factorization with areshaping process that adds nonlinearity to the approx-imation function.We deploy DeepThin as a plug-gablelibraryintegratedwithTensorFlowthatenablesuserstoseamlessly compress models at dif f erent granularities.We evaluate DeepThin on two state-of-the-art acous-tic models, TFKaldi and DeepSpeech, comparing it toprevious compression work (Pruning, HashNet, andRank Factorization), empirical limit study approaches,and hand-tuned models. For TFKaldi, our DeepThinnetworks show better word error rates (WER) thancompeting methods at practically all tested compres-sion rates, achieving an average of 60% relative im-provement over rank factorization, 57% over pruning,23% over hand-tuned same-size networks, and 6% overthe computationally expensive HashedNets. For Deep-Speech, DeepThin-compressed networks achieve bettertest loss than all other compression methods, reachinga 28% better result than rank factorization, 27% betterthan pruning, 20% better than hand-tuned same-sizenetworks, and 12% better than HashedNets.DeepThin also provide inference performance bene-f i ts in two ways: (1) by shrinking the application work-ing sets, allowing the model to f i t in a level of the∗ Work done while at Intel.arXiv, February, 2018cache/memory hierarchy where the original networkwas too large, and (2) by exploiting unique featuresof the technique to reuse many intermediate computa-tions, reducing the total compute operations necessary.We evaluate the performance of DeepThin inferenceacross three Haswell- and Broadwell-based platformswith varying cache sizes. Speedups range from 2X to14X, depending on the compression ratio and platformcache sizes.1 Introduction and MotivationIn recent years, machine learning algorithms have beenincreasingly used in consumer-facing products, suchas speech recognition in personal assistants. These al-gorithms rely on large weight matrices which encodethe relationships between dif f erent nodes in a network.Ideally, these algorithms would run directly on theclient devices such as Amazon Echo [ 20 ] and GoogleHome [ 14 ], which utilize them. Unfortunately, becausesuch devices are usually portable, low-power devices,running such expensive algorithms is infeasible due tothe signif i cant storage, performance, and energy limi-tations involved.To work around this problem, many developers haveresorted to executing the inference models on high-performance cloud servers and streaming the inputsand outputs of the model between client and server.This solution, however, introduces many issues, includ-ing high operational costs, use of large amounts of datatransfer on metered (mobile) networks, user privacyconcerns, and increased latency.Recent research has investigated methods of com-pressing models to sizes which can be ef f i ciently exe-cuted directly on the client device. Such compressionapproaches must reduce the model space requirementwithout signif i cantly impacting prediction accuracy,runtime performance, or engineering time. Our workbuilds on existing research in the area of low rank fac-torization. We develop a new compression method andlibrary, DeepThin, that:arXiv:1802.06944v1 [cs.LG] 20 Feb 2018