Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Decreasing Annotation Burden of Pairwise Comparisons with
  Human-in-the-Loop Sorting: Application in Medical Image Artifact Rating   Ranking by pairwise comparisons has shown improved reliability over ordinal
classification. However, as the annotations of pairwise comparisons scale
quadratically, this becomes less practical when the dataset is large. We
propose a method for reducing the number of pairwise comparisons required to
rank by a quantitative metric, demonstrating the effectiveness of the approach
in ranking medical images by image quality in this proof of concept study.
Using the medical image annotation software that we developed, we actively
subsample pairwise comparisons using a sorting algorithm with a human rater in
the loop. We find that this method substantially reduces the number of
comparisons required for a full ordinal ranking without compromising
inter-rater reliability when compared to pairwise comparisons without sorting.

14
The categories of this paper are: math-ph
13
The categories of this paper are: quant-ph
28
The categories of this paper are: alg-geom
11
The categories of this paper are: hep-lat
18
The categories of this paper are: econ
30
The categories of this paper are: q-alg
16
The categories of this paper are: stat
12
The categories of this paper are: nucl-th
36
The categories of this paper are: cmp-lg
26
The categories of this paper are: bayes-an
6
The categories of this paper are: astro-ph
33
The categories of this paper are: plasm-ph
9
The categories of this paper are: nlin
22
The categories of this paper are: chao-dyn
17
The categories of this paper are: q-fin
0
The categories of this paper are: hep-ph