LSTM Pose MachinesYue Luo 1 Jimmy Ren 1 Zhouxia Wang 1 Wenxiu Sun 1 Jinshan Pan 1 Jianbo Liu 1 Jiahao Pang 1 Liang Lin 1,21 SenseTime Research2 Sun Yat-sen University, China1 {luoyue, rensijie, wangzhouxia, sunwenxiu, panjinshan, liujianbo, pangjiahao, linliang}@sensetime.comAbstractWe observed that recent state-of-the-art results on sin-gle image human pose estimation were achieved by multi-stage Convolution Neural Networks (CNN). Notwithstand-ing the superior performance on static images, the applica-tion of these models on videos is not only computationallyintensive, italsosuffersfromperformancedegenerationandf l icking. Such suboptimal results are mainly attributed tothe inability of imposing sequential geometric consistency,handling severe image quality degradation (e.g. motionblur and occlusion) as well as the inability of capturing thetemporal correlation among video frames. In this paper, weproposed a novel recurrent network to tackle these prob-lems. We showed that if we were to impose the weight shar-ing scheme to the multi-stage CNN, it could be re-writtenas a Recurrent Neural Network (RNN). This property de-couples the relationship among multiple network stages andresults in signif i cantly faster speed in invoking the networkfor videos. It also enables the adoption of Long Short-TermMemory(LSTM)unitsbetweenvideoframes. Wefoundsuchmemory augmented RNN is very effective in imposing geo-metric consistency among frames. It also well handles in-put quality degradation in videos while successfully stabi-lizes the sequential outputs. The experiments showed thatour approach signif i cantly outperformed current state-of-the-art methods on two large-scale video pose estimationbenchmarks. We also explored the memory cells inside theLSTM and provided insights on why such mechanism wouldbenef i t the prediction for video-based pose estimations. 11. IntroductionEstimating joint locations of human bodies is a challeng-ing problem in computer vision which f i nds many real ap-plications in areas including augmented reality, animationand automatic photo editing. Previous methods [2, 6, 38]1 Code is publicly available at https://github.com/lawy623/LSTM_Pose_Machines.Figure 1. Comparison of results produced by Convolutional PoseMachine (CPM) [36] after setting the video as a series of static im-ages (Up) and our method (Down). Several problems occur duringpose estimation on videos: a) Errors and our correct results in es-timating symmetric joints. b) Errors and our correct results whenjoints are occluded. c) Flicking results and our results when thebody moves rapidly.mainly addressed this problem by well designed graphicalmodels. Newly developed approaches [5, 23, 36] achievedhigher performance with deep Convolutional Neural Net-works (CNN).Nevertheless, those state-of-the-art models were trainedon still images, limiting their performance on videos. Fig-ure 1 demonstrates some unsatisfactory situations. For in-stance, the lack of geometric consistency makes the pre-vious methods prone to making obvious errors. Mistakescaused by serious occlusion and large motion are not un-common as well. In addition, those models usually havea deep architecture and would be computationally very in-tensive for real-time applications. Therefore, a relativelylight-weight model is preferable if we want to deploy it in aarXiv:1712.06316v4 [cs.CV] 9 Mar 2018
