Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Evaluating Artificial Systems for Pairwise Ranking Tasks Sensitive to
  Individual Differences   Owing to the advancement of deep learning, artificial systems are now rival
to humans in several pattern recognition tasks, such as visual recognition of
object categories. However, this is only the case with the tasks for which
correct answers exist independent of human perception. There is another type of
tasks for which what to predict is human perception itself, in which there are
often individual differences. Then, there are no longer single "correct"
answers to predict, which makes evaluation of artificial systems difficult. In
this paper, focusing on pairwise ranking tasks sensitive to individual
differences, we propose an evaluation method. Given a ranking result for
multiple item pairs that is generated by an artificial system, our method
quantifies the probability that the same ranking result will be generated by
humans, and judges if it is distinguishable from human-generated results. We
introduce a probabilistic model of human ranking behavior, and present an
efficient computation method for the judgment. To estimate model parameters
accurately from small-size samples, we present a method that uses confidence
scores given by annotators for ranking each item pair. Taking as an example a
task of ranking image pairs according to material attributes of objects, we
demonstrate how the proposed method works.

31
The categories of this paper are: ao-sci
34
The categories of this paper are: chem-ph
9
The categories of this paper are: nlin
37
The categories of this paper are: supr-con
29
The categories of this paper are: funct-an
13
The categories of this paper are: quant-ph
11
The categories of this paper are: hep-lat
35
The categories of this paper are: mtrl-th
24
The categories of this paper are: dg-ga
6
The categories of this paper are: astro-ph
18
The categories of this paper are: econ
23
The categories of this paper are: solv-int
27
The categories of this paper are: comp-gas
28
The categories of this paper are: alg-geom
12
The categories of this paper are: nucl-th
21
The categories of this paper are: adap-org