Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: How is BERT surprised? Layerwise detection of linguistic anomalies   Transformer language models have shown remarkable ability in detecting when a
word is anomalous in context, but likelihood scores offer no information about
the cause of the anomaly. In this work, we use Gaussian models for density
estimation at intermediate layers of three language models (BERT, RoBERTa, and
XLNet), and evaluate our method on BLiMP, a grammaticality judgement benchmark.
In lower layers, surprisal is highly correlated to low token frequency, but
this correlation diminishes in upper layers. Next, we gather datasets of
morphosyntactic, semantic, and commonsense anomalies from psycholinguistic
studies; we find that the best performing model RoBERTa exhibits surprisal in
earlier layers when the anomaly is morphosyntactic than when it is semantic,
while commonsense anomalies do not exhibit surprisal at any intermediate layer.
These results suggest that language models employ separate mechanisms to detect
different types of linguistic anomalies.

24
The categories of this paper are: dg-ga
3
The categories of this paper are: physics
32
The categories of this paper are: atom-ph
33
The categories of this paper are: plasm-ph
9
The categories of this paper are: nlin
2
The categories of this paper are: math
7
The categories of this paper are: hep-th
13
The categories of this paper are: quant-ph
28
The categories of this paper are: alg-geom
6
The categories of this paper are: astro-ph
0
The categories of this paper are: hep-ph
35
The categories of this paper are: mtrl-th
23
The categories of this paper are: solv-int
20
The categories of this paper are: acc-phys
18
The categories of this paper are: econ
37
The categories of this paper are: supr-con