Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Evaluation Discrepancy Discovery: A Sentence Compression Case-study   Reliable evaluation protocols are of utmost importance for reproducible NLP
research. In this work, we show that sometimes neither metric nor conventional
human evaluation is sufficient to draw conclusions about system performance.
Using sentence compression as an example task, we demonstrate how a system can
game a well-established dataset to achieve state-of-the-art results. In
contrast with the results reported in previous work that showed correlation
between human judgements and metric scores, our manual analysis of
state-of-the-art system outputs demonstrates that high metric scores may only
indicate a better fit to the data, but not better outputs, as perceived by
humans.

15
The categories of this paper are: nucl-ex
13
The categories of this paper are: quant-ph
33
The categories of this paper are: plasm-ph
2
The categories of this paper are: math
19
The categories of this paper are: eess
9
The categories of this paper are: nlin
29
The categories of this paper are: funct-an
23
The categories of this paper are: solv-int
5
The categories of this paper are: gr-qc
30
The categories of this paper are: q-alg
28
The categories of this paper are: alg-geom
6
The categories of this paper are: astro-ph
3
The categories of this paper are: physics
17
The categories of this paper are: q-fin
7
The categories of this paper are: hep-th
27
The categories of this paper are: comp-gas