Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual
  Transfer with Multilingual Transformers   Massively multilingual transformers pretrained with language modeling
objectives (e.g., mBERT, XLM-R) have become a de facto default transfer
paradigm for zero-shot cross-lingual transfer in NLP, offering unmatched
transfer performance. Current downstream evaluations, however, verify their
efficacy predominantly in transfer settings involving languages with sufficient
amounts of pretraining data, and with lexically and typologically close
languages. In this work, we analyze their limitations and show that
cross-lingual transfer via massively multilingual transformers, much like
transfer via cross-lingual word embeddings, is substantially less effective in
resource-lean scenarios and for distant languages. Our experiments,
encompassing three lower-level tasks (POS tagging, dependency parsing, NER), as
well as two high-level semantic tasks (NLI, QA), empirically correlate transfer
performance with linguistic similarity between the source and target languages,
but also with the size of pretraining corpora of target languages. We also
demonstrate a surprising effectiveness of inexpensive few-shot transfer (i.e.,
fine-tuning on a few target-language instances after fine-tuning in the source)
across the board. This suggests that additional research efforts should be
invested to reach beyond the limiting zero-shot conditions.

4
The categories of this paper are: cond-mat
9
The categories of this paper are: nlin
30
The categories of this paper are: q-alg
0
The categories of this paper are: hep-ph
28
The categories of this paper are: alg-geom
20
The categories of this paper are: acc-phys
2
The categories of this paper are: math
37
The categories of this paper are: supr-con
27
The categories of this paper are: comp-gas
32
The categories of this paper are: atom-ph
3
The categories of this paper are: physics
7
The categories of this paper are: hep-th
31
The categories of this paper are: ao-sci
23
The categories of this paper are: solv-int
25
The categories of this paper are: patt-sol
24
The categories of this paper are: dg-ga