Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Implicit Regularization Towards Rank Minimization in ReLU Networks   We study the conjectured relationship between the implicit regularization in
neural networks, trained with gradient-based methods, and rank minimization of
their weight matrices. Previously, it was proved that for linear networks (of
depth 2 and vector-valued outputs), gradient flow (GF) w.r.t. the square loss
acts as a rank minimization heuristic. However, understanding to what extent
this generalizes to nonlinear networks is an open problem. In this paper, we
focus on nonlinear ReLU networks, providing several new positive and negative
results. On the negative side, we prove (and demonstrate empirically) that,
unlike the linear case, GF on ReLU networks may no longer tend to minimize
ranks, in a rather strong sense (even approximately, for "most" datasets of
size 2). On the positive side, we reveal that ReLU networks of sufficient depth
are provably biased towards low-rank solutions in several reasonable settings.

19
The categories of this paper are: eess
34
The categories of this paper are: chem-ph
32
The categories of this paper are: atom-ph
31
The categories of this paper are: ao-sci
4
The categories of this paper are: cond-mat
21
The categories of this paper are: adap-org
12
The categories of this paper are: nucl-th
26
The categories of this paper are: bayes-an
35
The categories of this paper are: mtrl-th
2
The categories of this paper are: math
0
The categories of this paper are: hep-ph
24
The categories of this paper are: dg-ga
23
The categories of this paper are: solv-int
22
The categories of this paper are: chao-dyn
17
The categories of this paper are: q-fin
30
The categories of this paper are: q-alg