Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese
  Pre-trained Language Models   Chinese pre-trained language models usually process text as a sequence of
characters, while ignoring more coarse granularity, e.g., words. In this work,
we propose a novel pre-training paradigm for Chinese -- Lattice-BERT, which
explicitly incorporates word representations along with characters, thus can
model a sentence in a multi-granularity manner. Specifically, we construct a
lattice graph from the characters and words in a sentence and feed all these
text units into transformers. We design a lattice position attention mechanism
to exploit the lattice structures in self-attention layers. We further propose
a masked segment prediction task to push the model to learn from rich but
redundant information inherent in lattices, while avoiding learning unexpected
tricks. Experiments on 11 Chinese natural language understanding tasks show
that our model can bring an average increase of 1.5% under the 12-layer
setting, which achieves new state-of-the-art among base-size models on the CLUE
benchmarks. Further analysis shows that Lattice-BERT can harness the lattice
structures, and the improvement comes from the exploration of redundant
information and multi-granularity representations. Our code will be available
at https://github.com/alibaba/pretrained-language-models/LatticeBERT.

33
The categories of this paper are: plasm-ph
5
The categories of this paper are: gr-qc
13
The categories of this paper are: quant-ph
23
The categories of this paper are: solv-int
2
The categories of this paper are: math
10
The categories of this paper are: q-bio
17
The categories of this paper are: q-fin
35
The categories of this paper are: mtrl-th
4
The categories of this paper are: cond-mat
20
The categories of this paper are: acc-phys
25
The categories of this paper are: patt-sol
19
The categories of this paper are: eess
32
The categories of this paper are: atom-ph
29
The categories of this paper are: funct-an
14
The categories of this paper are: math-ph
0
The categories of this paper are: hep-ph