Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: The Goldilocks zone: Towards better understanding of neural network loss
  landscapes   We explore the loss landscape of fully-connected and convolutional neural
networks using random, low-dimensional hyperplanes and hyperspheres. Evaluating
the Hessian, $H$, of the loss function on these hypersurfaces, we observe 1) an
unusual excess of the number of positive eigenvalues of $H$, and 2) a large
value of $\mathrm{Tr}(H) / ||H||$ at a well defined range of configuration
space radii, corresponding to a thick, hollow, spherical shell we refer to as
the \textit{Goldilocks zone}. We observe this effect for fully-connected neural
networks over a range of network widths and depths on MNIST and CIFAR-10
datasets with the $\mathrm{ReLU}$ and $\tanh$ non-linearities, and a similar
effect for convolutional networks. Using our observations, we demonstrate a
close connection between the Goldilocks zone, measures of local
convexity/prevalence of positive curvature, and the suitability of a network
initialization. We show that the high and stable accuracy reached when
optimizing on random, low-dimensional hypersurfaces is directly related to the
overlap between the hypersurface and the Goldilocks zone, and as a corollary
demonstrate that the notion of intrinsic dimension is initialization-dependent.
We note that common initialization techniques initialize neural networks in
this particular region of unusually high convexity/prevalence of positive
curvature, and offer a geometric intuition for their success. Furthermore, we
demonstrate that initializing a neural network at a number of points and
selecting for high measures of local convexity such as $\mathrm{Tr}(H) /
||H||$, number of positive eigenvalues of $H$, or low initial loss, leads to
statistically significantly faster training on MNIST. Based on our
observations, we hypothesize that the Goldilocks zone contains an unusually
high density of suitable initialization configurations.

18
The categories of this paper are: econ
8
The categories of this paper are: hep-ex
9
The categories of this paper are: nlin
37
The categories of this paper are: supr-con
17
The categories of this paper are: q-fin
32
The categories of this paper are: atom-ph
33
The categories of this paper are: plasm-ph
14
The categories of this paper are: math-ph
5
The categories of this paper are: gr-qc
31
The categories of this paper are: ao-sci
22
The categories of this paper are: chao-dyn
15
The categories of this paper are: nucl-ex
24
The categories of this paper are: dg-ga
36
The categories of this paper are: cmp-lg
29
The categories of this paper are: funct-an
30
The categories of this paper are: q-alg