Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Delve into the Performance Degradation of Differentiable Architecture
  Search   Differentiable architecture search (DARTS) is widely considered to be easy to
overfit the validation set which leads to performance degradation. We first
employ a series of exploratory experiments to verify that neither high-strength
architecture parameters regularization nor warmup training scheme can
effectively solve this problem. Based on the insights from the experiments, we
conjecture that the performance of DARTS does not depend on the well-trained
supernet weights and argue that the architecture parameters should be trained
by the gradients which are obtained in the early stage rather than the final
stage of training. This argument is then verified by exchanging the learning
rate schemes of weights and parameters. Experimental results show that the
simple swap of the learning rates can effectively solve the degradation and
achieve competitive performance. Further empirical evidence suggests that the
degradation is not a simple problem of the validation set overfitting but
exhibit some links between the degradation and the operation selection bias
within bilevel optimization dynamics. We demonstrate the generalization of this
bias and propose to utilize this bias to achieve an operation-magnitude-based
selective stop.

34
The categories of this paper are: chem-ph
17
The categories of this paper are: q-fin
30
The categories of this paper are: q-alg
10
The categories of this paper are: q-bio
22
The categories of this paper are: chao-dyn
16
The categories of this paper are: stat
25
The categories of this paper are: patt-sol
8
The categories of this paper are: hep-ex
27
The categories of this paper are: comp-gas
35
The categories of this paper are: mtrl-th
24
The categories of this paper are: dg-ga
7
The categories of this paper are: hep-th
36
The categories of this paper are: cmp-lg
11
The categories of this paper are: hep-lat
23
The categories of this paper are: solv-int
12
The categories of this paper are: nucl-th