Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Is the Number of Trainable Parameters All That Actually Matters?   Recent work has identified simple empirical scaling laws for language models,
linking compute budget, dataset size, model size, and autoregressive modeling
loss. The validity of these simple power laws across orders of magnitude in
model scale provides compelling evidence that larger models are also more
capable models. However, scaling up models under the constraints of hardware
and infrastructure is no easy feat, and rapidly becomes a hard and expensive
engineering problem. We investigate ways to tentatively cheat scaling laws, and
train larger models for cheaper. We emulate an increase in effective
parameters, using efficient approximations: either by doping the models with
frozen random parameters, or by using fast structured transforms in place of
dense linear layers. We find that the scaling relationship between test loss
and compute depends only on the actual number of trainable parameters; scaling
laws cannot be deceived by spurious parameters.

33
The categories of this paper are: plasm-ph
11
The categories of this paper are: hep-lat
26
The categories of this paper are: bayes-an
19
The categories of this paper are: eess
35
The categories of this paper are: mtrl-th
0
The categories of this paper are: hep-ph
20
The categories of this paper are: acc-phys
36
The categories of this paper are: cmp-lg
21
The categories of this paper are: adap-org
25
The categories of this paper are: patt-sol
12
The categories of this paper are: nucl-th
9
The categories of this paper are: nlin
3
The categories of this paper are: physics
15
The categories of this paper are: nucl-ex
7
The categories of this paper are: hep-th
2
The categories of this paper are: math