Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: LocoProp: Enhancing BackProp via Local Loss Optimization   Second-order methods have shown state-of-the-art performance for optimizing
deep neural networks. Nonetheless, their large memory requirement and high
computational complexity, compared to first-order methods, hinder their
versatility in a typical low-budget setup. This paper introduces a general
framework of layerwise loss construction for multilayer neural networks that
achieves a performance closer to second-order methods while utilizing
first-order optimizers only. Our methodology lies upon a three-component loss,
target, and regularizer combination, for which altering each component results
in a new update rule. We provide examples using squared loss and layerwise
Bregman divergences induced by the convex integral functions of various
transfer functions. Our experiments on benchmark models and datasets validate
the efficacy of our new approach, reducing the gap between first-order and
second-order optimizers.

14
The categories of this paper are: math-ph
4
The categories of this paper are: cond-mat
16
The categories of this paper are: stat
30
The categories of this paper are: q-alg
11
The categories of this paper are: hep-lat
24
The categories of this paper are: dg-ga
28
The categories of this paper are: alg-geom
22
The categories of this paper are: chao-dyn
25
The categories of this paper are: patt-sol
19
The categories of this paper are: eess
12
The categories of this paper are: nucl-th
0
The categories of this paper are: hep-ph
33
The categories of this paper are: plasm-ph
36
The categories of this paper are: cmp-lg
6
The categories of this paper are: astro-ph
31
The categories of this paper are: ao-sci