Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Systematic Analysis of Experiment Precision Measures and Methods for
  Experiments Comparison   The notion of experiment precision quantifies the variance of user ratings in
a subjective experiment. Although there exist measures assessing subjective
experiment precision, there are no systematic analyses of these measures
available in the literature. To the best of our knowledge, there is also no
systematic framework in the Multimedia Quality Assessment (MQA) field for
comparing subjective experiments in terms of their precision. Therefore, the
main idea of this paper is to propose a framework for comparing subjective
experiments in the field of MQA based on appropriate experiment precision
measures. We put forward three experiment precision measures and three related
experiment precision comparison methods. We systematically analyse the
performance of the measures and methods proposed. We do so both through a
stimulation study (varying user rating variance and bias) and by using data
from three real-world Quality of Experience (QoE) subjective experiments. In
the simulation study we focus on crowdsourcing QoE experiments, since they are
known to generate ratings with higher variance and bias, when compared to
traditional subjective experiment methodologies. We conclude that our proposed
measures and related comparison methods properly capture experiment precision
(both when tested on simulated and real-world data). One of the measures also
proves capable of dealing with even significantly biased responses. We believe
our experiment precision assessment framework will help compare different
subjective experiment methodologies. For example, it may help decide which
methodology results in more precise user ratings. This may potentially inform
future standardisation activities.

0
The categories of this paper are: hep-ph
15
The categories of this paper are: nucl-ex
21
The categories of this paper are: adap-org
36
The categories of this paper are: cmp-lg
19
The categories of this paper are: eess
37
The categories of this paper are: supr-con
3
The categories of this paper are: physics
27
The categories of this paper are: comp-gas
2
The categories of this paper are: math
11
The categories of this paper are: hep-lat
14
The categories of this paper are: math-ph
23
The categories of this paper are: solv-int
7
The categories of this paper are: hep-th
13
The categories of this paper are: quant-ph
35
The categories of this paper are: mtrl-th
22
The categories of this paper are: chao-dyn