Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Training Stronger Baselines for Learning to Optimize   Learning to optimize (L2O) has gained increasing attention since classical
optimizers require laborious problem-specific design and hyperparameter tuning.
However, there is a gap between the practical demand and the achievable
performance of existing L2O models. Specifically, those learned optimizers are
applicable to only a limited class of problems, and often exhibit instability.
With many efforts devoted to designing more sophisticated L2O models, we argue
for another orthogonal, under-explored theme: the training techniques for those
L2O models. We show that even the simplest L2O model could have been trained
much better. We first present a progressive training scheme to gradually
increase the optimizer unroll length, to mitigate a well-known L2O dilemma of
truncation bias (shorter unrolling) versus gradient explosion (longer
unrolling). We further leverage off-policy imitation learning to guide the L2O
learning, by taking reference to the behavior of analytical optimizers. Our
improved training techniques are plugged into a variety of state-of-the-art L2O
models, and immediately boost their performance, without making any change to
their model structures. Especially, by our proposed techniques, an earliest and
simplest L2O model can be trained to outperform the latest complicated L2O
models on a number of tasks. Our results demonstrate a greater potential of L2O
yet to be unleashed, and urge to rethink the recent progress. Our codes are
publicly available at: https://github.com/VITA-Group/L2O-Training-Techniques.

35
The categories of this paper are: mtrl-th
34
The categories of this paper are: chem-ph
32
The categories of this paper are: atom-ph
13
The categories of this paper are: quant-ph
5
The categories of this paper are: gr-qc
6
The categories of this paper are: astro-ph
14
The categories of this paper are: math-ph
37
The categories of this paper are: supr-con
19
The categories of this paper are: eess
7
The categories of this paper are: hep-th
21
The categories of this paper are: adap-org
22
The categories of this paper are: chao-dyn
30
The categories of this paper are: q-alg
36
The categories of this paper are: cmp-lg
27
The categories of this paper are: comp-gas
24
The categories of this paper are: dg-ga