Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Scaling Laws for Transfer   We study empirical scaling laws for transfer learning between distributions
in an unsupervised, fine-tuning setting. When we train increasingly large
neural networks from-scratch on a fixed-size dataset, they eventually become
data-limited and stop improving in performance (cross-entropy loss). When we do
the same for models pre-trained on a large language dataset, the slope in
performance gains is merely reduced rather than going to zero. We calculate the
effective data "transferred" from pre-training by determining how much data a
transformer of the same size would have required to achieve the same loss when
training from scratch. In other words, we focus on units of data while holding
everything else fixed. We find that the effective data transferred is described
well in the low data regime by a power-law of parameter count and fine-tuning
dataset size. We believe the exponents in these power-laws correspond to
measures of the generality of a model and proximity of distributions (in a
directed rather than symmetric sense). We find that pre-training effectively
multiplies the fine-tuning dataset size. Transfer, like overall performance,
scales predictably in terms of parameters, data, and compute.

33
The categories of this paper are: plasm-ph
9
The categories of this paper are: nlin
19
The categories of this paper are: eess
31
The categories of this paper are: ao-sci
28
The categories of this paper are: alg-geom
20
The categories of this paper are: acc-phys
37
The categories of this paper are: supr-con
11
The categories of this paper are: hep-lat
32
The categories of this paper are: atom-ph
0
The categories of this paper are: hep-ph
4
The categories of this paper are: cond-mat
3
The categories of this paper are: physics
10
The categories of this paper are: q-bio
30
The categories of this paper are: q-alg
18
The categories of this paper are: econ
8
The categories of this paper are: hep-ex