Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Exposing Shallow Heuristics of Relation Extraction Models with Challenge
  Data   The process of collecting and annotating training data may introduce
distribution artifacts which may limit the ability of models to learn correct
generalization behavior. We identify failure modes of SOTA relation extraction
(RE) models trained on TACRED, which we attribute to limitations in the data
annotation process. We collect and annotate a challenge-set we call Challenging
RE (CRE), based on naturally occurring corpus examples, to benchmark this
behavior. Our experiments with four state-of-the-art RE models show that they
have indeed adopted shallow heuristics that do not generalize to the
challenge-set data. Further, we find that alternative question answering
modeling performs significantly better than the SOTA models on the
challenge-set, despite worse overall TACRED performance. By adding some of the
challenge data as training examples, the performance of the model improves.
Finally, we provide concrete suggestion on how to improve RE data collection to
alleviate this behavior.

25
The categories of this paper are: patt-sol
30
The categories of this paper are: q-alg
32
The categories of this paper are: atom-ph
15
The categories of this paper are: nucl-ex
23
The categories of this paper are: solv-int
7
The categories of this paper are: hep-th
27
The categories of this paper are: comp-gas
13
The categories of this paper are: quant-ph
24
The categories of this paper are: dg-ga
0
The categories of this paper are: hep-ph
14
The categories of this paper are: math-ph
35
The categories of this paper are: mtrl-th
22
The categories of this paper are: chao-dyn
3
The categories of this paper are: physics
34
The categories of this paper are: chem-ph
36
The categories of this paper are: cmp-lg