Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: The Gap on GAP: Tackling the Problem of Differing Data Distributions in
  Bias-Measuring Datasets   Diagnostic datasets that can detect biased models are an important
prerequisite for bias reduction within natural language processing. However,
undesired patterns in the collected data can make such tests incorrect. For
example, if the feminine subset of a gender-bias-measuring coreference
resolution dataset contains sentences with a longer average distance between
the pronoun and the correct candidate, an RNN-based model may perform worse on
this subset due to long-term dependencies. In this work, we introduce a
theoretically grounded method for weighting test samples to cope with such
patterns in the test data. We demonstrate the method on the GAP dataset for
coreference resolution. We annotate GAP with spans of all personal names and
show that examples in the female subset contain more personal names and a
longer distance between pronouns and their referents, potentially affecting the
bias score in an undesired way. Using our weighting method, we find the set of
weights on the test instances that should be used for coping with these
correlations, and we re-evaluate 16 recently released coreference models.

32
The categories of this paper are: atom-ph
34
The categories of this paper are: chem-ph
14
The categories of this paper are: math-ph
5
The categories of this paper are: gr-qc
17
The categories of this paper are: q-fin
33
The categories of this paper are: plasm-ph
0
The categories of this paper are: hep-ph
30
The categories of this paper are: q-alg
19
The categories of this paper are: eess
27
The categories of this paper are: comp-gas
13
The categories of this paper are: quant-ph
36
The categories of this paper are: cmp-lg
22
The categories of this paper are: chao-dyn
37
The categories of this paper are: supr-con
29
The categories of this paper are: funct-an
8
The categories of this paper are: hep-ex