Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On the Effect of Pretraining Corpora on In-context Learning by a
  Large-scale Language Model   Many recent studies on large-scale language models have reported successful
in-context zero- and few-shot learning ability. However, the in-depth analysis
of when in-context learning occurs is still lacking. For example, it is unknown
how in-context learning performance changes as the training corpus varies.
Here, we investigate the effects of the source and size of the pretraining
corpus on in-context learning in HyperCLOVA, a Korean-centric GPT-3 model. From
our in-depth investigation, we introduce the following observations: (1)
in-context learning performance heavily depends on the corpus domain source,
and the size of the pretraining corpus does not necessarily determine the
emergence of in-context learning, (2) in-context learning ability can emerge
when a language model is trained on a combination of multiple corpora, even
when each corpus does not result in in-context learning on its own, (3)
pretraining with a corpus related to a downstream task does not always
guarantee the competitive in-context learning performance of the downstream
task, especially in the few-shot setting, and (4) the relationship between
language modeling (measured in perplexity) and in-context learning does not
always correlate: e.g., low perplexity does not always imply high in-context
few-shot learning performance.

13
The categories of this paper are: quant-ph
19
The categories of this paper are: eess
21
The categories of this paper are: adap-org
12
The categories of this paper are: nucl-th
23
The categories of this paper are: solv-int
36
The categories of this paper are: cmp-lg
33
The categories of this paper are: plasm-ph
20
The categories of this paper are: acc-phys
2
The categories of this paper are: math
14
The categories of this paper are: math-ph
25
The categories of this paper are: patt-sol
37
The categories of this paper are: supr-con
4
The categories of this paper are: cond-mat
0
The categories of this paper are: hep-ph
34
The categories of this paper are: chem-ph
35
The categories of this paper are: mtrl-th