Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: "Is Whole Word Masking Always Better for Chinese BERT?": Probing on
  Chinese Grammatical Error Correction   Whole word masking (WWM), which masks all subwords corresponding to a word at
once, makes a better English BERT model. For the Chinese language, however,
there is no subword because each token is an atomic character. The meaning of a
word in Chinese is different in that a word is a compositional unit consisting
of multiple characters. Such difference motivates us to investigate whether WWM
leads to better context understanding ability for Chinese BERT. To achieve
this, we introduce two probing tasks related to grammatical error correction
and ask pretrained models to revise or insert tokens in a masked language
modeling manner. We construct a dataset including labels for 19,075 tokens in
10,448 sentences. We train three Chinese BERT models with standard
character-level masking (CLM), WWM, and a combination of CLM and WWM,
respectively. Our major findings are as follows: First, when one character
needs to be inserted or replaced, the model trained with CLM performs the best.
Second, when more than one character needs to be handled, WWM is the key to
better performance. Finally, when being fine-tuned on sentence-level downstream
tasks, models trained with different masking strategies perform comparably.

21
The categories of this paper are: adap-org
10
The categories of this paper are: q-bio
33
The categories of this paper are: plasm-ph
28
The categories of this paper are: alg-geom
20
The categories of this paper are: acc-phys
27
The categories of this paper are: comp-gas
37
The categories of this paper are: supr-con
18
The categories of this paper are: econ
8
The categories of this paper are: hep-ex
32
The categories of this paper are: atom-ph
5
The categories of this paper are: gr-qc
26
The categories of this paper are: bayes-an
13
The categories of this paper are: quant-ph
19
The categories of this paper are: eess
14
The categories of this paper are: math-ph
22
The categories of this paper are: chao-dyn