C OLLAPSE OF DEEP AND NARROW NEURAL NETSLu LuDivision of Applied MathematicsBrown UniversityProvidence, RI 02912, USAlu lu 1@brown.eduYanhui SuCollege of Mathematics and Computer ScienceFuzhou UniversityFuzhou, Fujian 350116, Chinasuyh@fzu.edu.cnGeorge Em KarniadakisDivision of Applied MathematicsBrown UniversityProvidence, RI 02912, USAgeorge karniadakis@brown.eduA BSTRACTRecent theoretical work has demonstrated that deep neural networks have supe-rior performance over shallow networks, but their training is more diff i cult, e.g.,they suffer from the vanishing gradient problem. This problem can be typicallyresolved by the rectif i ed linear unit (ReLU) activation. However, here we showthat even for such activation, deep and narrow neural networks will converge toerroneous mean or median states of the target function depending on the loss withhigh probability. We demonstrate this collapse of deep and narrow neural net-works both numerically and theoretically, and provide estimates of the probabilityof collapse. We also construct a diagram of a safe region of designing neural net-works that avoid the collapse to erroneous states. Finally, we examine differentways of initialization and normalization that may avoid the collapse problem.1 I NTRODUCTIONThe best-known universal approximation theorems of neural networks (NNs) were obtained almostthree decades ago by Cybenko (1989) and Hornik et al. (1989), stating that every measurable func-tion can be approximated accurately by a single-hidden-layer neural network, i.e., a shallow neuralnetwork. Although powerful, these results do not provide any information on the required size of aneural network to achieve a pre-specif i ed accuracy. In Barron (1993), the author analyzed the sizeof a neural network to approximate functions using Fourier transforms. Subsequently, in Mhaskar(1996), the authors considered optimal approximations of smooth and analytic functions in shal-low networks, and demonstrated that ? −d/n neurons can uniformly approximate any C n -functionon a compact set in R d with error ?. This is an interesting result and it shows that to approximatea three-dimensional function with accuracy 10 −6 we need to design a NN with 10 18 neurons fora C 1 function, but for a very smooth function, e.g., C 6 , we only need 1000 neurons. In the last15 years, deep neural networks (i.e., networks with a large number of layers) have been used veryeffectively in diverse applications, such as image classif i cation (Krizhevsky et al., 2012), speechrecognition (Hinton et al., 2012), game intelligence (Silver et al., 2016), regression (Lagaris et al.,1997; 2000), etc.After some initial debate, at the present time, it seems that deep NNs perform better than shallowNNs of comparable size, e.g., a 3-layer NN with 10 neurons per layer may be a better approximatorthan a 1-layer NN with 30 neurons. From the approximation point of view, there are several theo-retical results to explain this superior performance. In Eldan & Shamir (2016), the authors showedthat a simple approximately radial function can be approximated by a small 3-layer feed-forwardNN, but it cannot be approximated by any 2-layer network with the same accuracy irrespective ofthe activation function, unless its width is exponential in the dimension (see Mhaskar et al. (2017);Mhaskar & Poggio (2016); Delalleau & Bengio (2011); Poggio et al. (2017) for further discussions).In Liang & Srikant (2017) (see also Yarotsky (2017)), the authors claimed that for ?-approximation1arXiv:1808.04947v1 [stat.ML] 15 Aug 2018
