Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Subword Pooling Makes a Difference   Contextual word-representations became a standard in modern natural language
processing systems. These models use subword tokenization to handle large
vocabularies and unknown words. Word-level usage of such systems requires a way
of pooling multiple subwords that correspond to a single word. In this paper we
investigate how the choice of subword pooling affects the downstream
performance on three tasks: morphological probing, POS tagging and NER, in 9
typologically diverse languages. We compare these in two massively multilingual
models, mBERT and XLM-RoBERTa. For morphological tasks, the widely used `choose
the first subword' is the worst strategy and the best results are obtained by
using attention over the subwords. For POS tagging both of these strategies
perform poorly and the best choice is to use a small LSTM over the subwords.
The same strategy works best for NER and we show that mBERT is better than
XLM-RoBERTa in all 9 languages. We publicly release all code, data and the full
result tables at \url{https://github.com/juditacs/subword-choice}.

14
The categories of this paper are: math-ph
7
The categories of this paper are: hep-th
36
The categories of this paper are: cmp-lg
4
The categories of this paper are: cond-mat
11
The categories of this paper are: hep-lat
9
The categories of this paper are: nlin
23
The categories of this paper are: solv-int
10
The categories of this paper are: q-bio
26
The categories of this paper are: bayes-an
35
The categories of this paper are: mtrl-th
16
The categories of this paper are: stat
8
The categories of this paper are: hep-ex
19
The categories of this paper are: eess
25
The categories of this paper are: patt-sol
15
The categories of this paper are: nucl-ex
22
The categories of this paper are: chao-dyn