C-LSTM: Enabling Ef f icient LSTM using Structured CompressionTechniques on FPGAsShuo Wang 1,+ , Zhe Li 2,+ , Caiwen Ding 2,+ , Bo Yuan 3 , Qinru Qiu 2 , Yanzhi Wang 2 and Yun Liang 1,∗+ These authors contributed equally1 Center for Energy-Ef f i cient Computing & Applications (CECA), School of EECS, Peking University, China2 Dept. of Electrical Engineering & Computer Science, Syracuse University, Syracuse, NY, USA3 Dept. of Electrical Engineering, City University of New York, NY, USA1 {shvowang,ericlyun}@pku.edu.cn, 2 {zli89,cading,qiqiu,ywang393}@syr.edu, 3 byuan@ccny.cuny.eduABSTRACTRecently, signif i cant accuracy improvement has been achieved foracoustic recognition systems by increasing the model size of LongShort-Term Memory (LSTM) networks. Unfortunately, the ever-increasingsizeofLSTMmodelleadstoinef f i cientdesignsonFPGAsduetothelimitedon-chipresources.Thepreviousworkproposestouseapruningbasedcompressiontechniquetoreducethemodelsizeand thus speedups the inference on FPGAs. However, the randomnature of the pruning technique transforms the dense matricesof the model to highly unstructured sparse ones, which leads tounbalanced computation and irregular memory accesses and thushurts the overall performance and energy ef f i ciency.In contrast, we propose to use a structured compression tech-nique which could not only reduce the LSTM model size but alsoeliminate the irregularities of computation and memory accesses.This approach employs block-circulant instead of sparse matricesto compress weight matrices and reduces the storage requirementfrom ?(k 2 ) to ?(k) . Fast Fourier Transform algorithm is utilizedto further accelerate the inference by reducing the computationalcomplexity from ?(k 2 ) to ?(klogk) . The datapath and activationfunctions are quantized as 16-bit to improve the resource utilization.More importantly, we propose a comprehensive framework calledC-LSTM to automatically optimize and implement a wide range ofLSTM variants on FPGAs. According to the experimental results,C-LSTM achieves up to 18.8X and 33.5X gains for performance andenergy ef f i ciency compared with the state-of-the-art LSTM imple-mentation under the same experimental setup, and the accuracydegradation is very small.KEYWORDSFPGA; RNNs; LSTM; compression; block-circulant matrix; FFT∗ Corresponding author.Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor prof i t or commercial advantage and that copies bear this notice and the full citationon the f i rst page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specif i c permission and/or afee. Request permissions from permissions@acm.org.FPGA’18, Feb. 25–27, 2018, Monterey, CA, USA© 2018 Association for Computing Machinery.ACM ISBN 978-1-4503-5614-5/18/02...$15.00https://doi.org/10.1145/3174243.3174253ACM Reference Format:Shuo Wang 1,+ , Zhe Li 2,+ , Caiwen Ding 2,+ , Bo Yuan 3 , Qinru Qiu 2 , YanzhiWang 2 andYunLiang 1, .2018.C-LSTM:EnablingEf f i cientLSTMusingStruc-tured Compression Techniques on FPGAs. In FPGA’18: 2018 ACM/SIGDAInternational Symposium on Field-Programmable Gate Arrays, February 25–27, 2018, Monterey, CA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3174243.31742531 INTRODUCTIONRecurrent neural networks (RNNs) represent an important classof neural networks that contain cycles to carry information acrossneurons while reading inputs. Long Short-Term Memory (LSTM),one of the most popular types of RNNs, achieves great success inthe domains such as speech recognition, machine translation, sceneanalysis, etc. [25]. However, the signif i cant recognition accuracyimprovement comes at the cost of increased computational com-plexity of larger model size [9]. Therefore, customized hardwareacceleration is increasingly important for LSTMs, as exemplif i edby recent works on employing GPUs [5, 17], FPGAs [13, 16] andASICs [7] as accelerators to speedup LSTMs.Among the numerous platforms, FPGA has emerged as a promis-ing solution for hardware acceleration as it provides customizedhardware performance with f l exible reconf i gurability. By creatingdedicated pipelines, parallel processing units, customized bit width,and etc., application designers can accelerate many workloads byorders of magnitude using FPGAs [24]. More importantly, High-level Synthesis (HLS) has greatly lowered the programming hurdleof FPGAs and improved the productivity by raising the program-ming abstraction from tedious RTL to high-level languages such asC/C++ [4] and OpenCL [26].While the benef i ts of FPGAs is clear, it is still challenging to de-sign ef f i cient designs for LSTMs on FPGAs mainly for two reasons.On one hand, the capacity of the FPGA on-chip memory (a few ortens of Mb on-chip memory) is usually not large enough to storeall the weight matrices of a standard LSTM inference model (e.g.hundreds of Mb). Although the previous work ESE [13] proposesto use the parameter pruning based compression technique to com-press the dense weight matrices in the LSTM model into sparseones, the sparse matrices need extra storage and processing unitsto store and decode the indices of the non-zero data, respectively.The skewed distribution of the data is likely to cause unbalancedworkloads among parallel compute units. Therefore, the benef i ts ofunstructured model compression is diminished by the sparsity ofweight matrices. On the other hand, the computational complexityamong the operators of the LSTMs is highly skewed and the dataarXiv:1803.06305v1 [cs.DC] 14 Mar 2018
