Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: You should evaluate your language model on marginal likelihood over
  tokenisations   Neural language models typically tokenise input text into sub-word units to
achieve an open vocabulary. The standard approach is to use a single canonical
tokenisation at both train and test time. We suggest that this approach is
unsatisfactory and may bottleneck our evaluation of language model performance.
Using only the one-best tokenisation ignores tokeniser uncertainty over
alternative tokenisations, which may hurt model out-of-domain performance.
  In this paper, we argue that instead, language models should be evaluated on
their marginal likelihood over tokenisations. We compare different estimators
for the marginal likelihood based on sampling, and show that it is feasible to
estimate the marginal likelihood with a manageable number of samples. We then
evaluate pretrained English and German language models on both the
one-best-tokenisation and marginal perplexities, and show that the marginal
perplexity can be significantly better than the one best, especially on
out-of-domain data. We link this difference in perplexity to the tokeniser
uncertainty as measured by tokeniser entropy. We discuss some implications of
our results for language model training and evaluation, particularly with
regard to tokenisation robustness.

27
The categories of this paper are: comp-gas
4
The categories of this paper are: cond-mat
22
The categories of this paper are: chao-dyn
13
The categories of this paper are: quant-ph
2
The categories of this paper are: math
8
The categories of this paper are: hep-ex
6
The categories of this paper are: astro-ph
18
The categories of this paper are: econ
17
The categories of this paper are: q-fin
34
The categories of this paper are: chem-ph
23
The categories of this paper are: solv-int
15
The categories of this paper are: nucl-ex
10
The categories of this paper are: q-bio
36
The categories of this paper are: cmp-lg
32
The categories of this paper are: atom-ph
0
The categories of this paper are: hep-ph