Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: What do we Really Know about State of the Art NER?   Named Entity Recognition (NER) is a well researched NLP task and is widely
used in real world NLP scenarios. NER research typically focuses on the
creation of new ways of training NER, with relatively less emphasis on
resources and evaluation. Further, state of the art (SOTA) NER models, trained
on standard datasets, typically report only a single performance measure
(F-score) and we don't really know how well they do for different entity types
and genres of text, or how robust are they to new, unseen entities. In this
paper, we perform a broad evaluation of NER using a popular dataset, that takes
into consideration various text genres and sources constituting the dataset at
hand. Additionally, we generate six new adversarial test sets through small
perturbations in the original test set, replacing select entities while
retaining the context. We also train and test our models on randomly generated
train/dev/test splits followed by an experiment where the models are trained on
a select set of genres but tested genres not seen in training. These
comprehensive evaluation strategies were performed using three SOTA NER models.
Based on our results, we recommend some useful reporting practices for NER
researchers, that could help in providing a better understanding of a SOTA
model's performance in future.

15
The categories of this paper are: nucl-ex
5
The categories of this paper are: gr-qc
29
The categories of this paper are: funct-an
27
The categories of this paper are: comp-gas
7
The categories of this paper are: hep-th
0
The categories of this paper are: hep-ph
23
The categories of this paper are: solv-int
8
The categories of this paper are: hep-ex
10
The categories of this paper are: q-bio
11
The categories of this paper are: hep-lat
17
The categories of this paper are: q-fin
2
The categories of this paper are: math
37
The categories of this paper are: supr-con
33
The categories of this paper are: plasm-ph
24
The categories of this paper are: dg-ga
35
The categories of this paper are: mtrl-th