Deep Quaternion NetworksChase J. GaudetSchool of Computing & InformaticsUniversity of Lousiana at LafayetteLafayette, USAcjg7182@louisiana.eduAnthony S. MaidaSchool of Computing & InformaticsUniversity of Lousiana at LafayetteLafayette, USAmaida@louisiana.eduAbstract—The f i eld of deep learning has seen signif i cantadvancement in recent years. However, much of the existing workhas been focused on real-valued numbers. Recent work has shownthat a deep learning system using the complex numbers can bedeeper for a f i xed parameter budget compared to its real-valuedcounterpart. In this work, we explore the benef i ts of generalizingone step further into the hyper-complex numbers, quaternionsspecif i cally, and provide the architecture components neededto build deep quaternion networks. We develop the theoreticalbasis by reviewing quaternion convolutions, developing a novelquaternion weight initialization scheme, and developing novelalgorithms for quaternion batch-normalization. These pieces aretested in a classif i cation model by end-to-end training on theCIFAR-10 and CIFAR-100 data sets and a segmentation modelby end-to-end training on the KITTI Road Segmentation dataset. These quaternion networks show improved convergencecompared to real-valued and complex-valued networks, especiallyon the segmentation task, while having fewer parameters.Index Terms—quaternion, complex, neural networks, deeplearningI. I NTRODUCTIONThere have been many advances in deep neural networkarchitectures in the past few years. One such improvementis a normalization technique called batch normalization [1]that standardizes the activations of layers inside a networkusing minibatch statistics. It has been shown to regularize thenetwork as well as provide faster and more stable training.Another improvement comes from architectures that add socalled shortcut paths to the network. These shortcut pathsconnect later layers to earlier layers typically, which allows forthe stronger gradients to propagate to the earlier layers. Thismethod can be seen in Highway Networks [2] and ResidualNetworks [3]. Other work has been done to f i nd new activationfunctions with more desirable properties. One example isthe exponential linear unit (ELU) [4], which attempts tokeep activations standardized. All of the above methods arecombating the vanishing gradient problem [5] that plaguesdeep architectures. With solutions to this problem appearingit is only natural to move to a system that will allow one toconstruct deeper architectures with as low a parameter cost aspossible.Other work in this area has explored the use of complexand hyper-complex numbers, which are a generalization ofthe complex, such as quaternions. Using complex numbersin recurrent neural networks (RNNs) has been shown toincrease learning speed and provide a more noise robustmemory retrieval mechanism [6]–[8]. The f i rst formulation ofcomplex batch normalization and complex weight initializationis presented by [9] where they achieve some state of the artresults on the MusicNet data set. Hyper-complex numbersare less explored in neural networks, but have seen use inmanual image and signal processing techniques [10]–[12].Examples of using quaternion values in networks is mostlylimited to architectures that take in quaternion inputs or predictquaternion outputs, but do not have quaternion weight values[13], [14]. There are some more recent examples of buildingmodels that use quaternions represented as real-values. In [15]they used a quaternion multi-layer perceptron (QMLP) fordocument understanding and [16] uses a similar approach inprocessing multi-dimensional signals.Building on [9] our contribution in this paper is to formulateand implement quaternion convolution, batch normalization,and weight initialization1 . There arises some diff i culty overcomplex batch normalization that we had to overcome as theiris no analytic form for our inverse square root matrix.II. M OTIVATION AND R ELATED W ORKThe ability of quaternions to effectively represent spatialtransformations and analyze multi-dimensional signals makesthem promising for applications in artif i cial intelligence.One common use of quaternions is for representing rotationinto a more compact form. PoseNet [14] used a quaternionas the target output in their model where the goal was torecover the 6−DOF camera pose from a single RGB image.The ability to encode rotations may make a quaternion networkmore robust to rotational variance.Quaternion representation has also been used in signalprocessing. The amount of information in the phase of animage has been shown to be suff i cient to recover the majorityof information encoded in its magnitude by Oppenheim andLin [17]. The phase also encodes information such as shapes,edges, and orientations. Quaternions can be represented as a2 x 2 matrix of complex numbers, which gives them a groupof phases potentially holding more information compared to asingle phase.Bulow and Sommer [12] used the higher complexity repre-sentation of quaternions by extending Gabor’s complex signal1 Source code located at https://github.com/gaudetcj/DeepQuaternionNetworksarXiv:1712.04604v3 [cs.NE] 29 Jul 2018
